I’ve been looking around for a new host due to the increase in bandwidth usage. Oxilion has been found willing to sponsor the forum which I greatly appreciate.
This move provides an opportunity to move to the latest forum software version. The non-commercial OpenSource successor of PunBB is FluxBB which allows easy migration of changes like the external authentication against the OpenStreetMap user database. The new forum software should support UTF-8 out of the box and has some extra features too.
The upgrade will go as follows:
Install the forum on the new host [done]
Add necessary changes to the forum software (remote authentication) [done]
The current forum will become read-only
A database dump is created
The database dump is loaded on the new host
The database is converted to the new forum software version
Yes, that’s why I used the word “also”…
I would appreciate if there forum has his own userlist to remain accessible when OSM.org is offline. This list should be automaticaly updated if ther user changes something in his usersettings on OSM.org. The user should not be able to change his username / password inside the forum.
Yes, I’ve been thinking about expanding the remote authentication with storing the username and password in the forum db as well (the username is already stored ofcourse). When the local forum db authentication fails the script could then try to authenticate against the OSM API. So users who do not change their password should be able to login even when the OSM API is offline but only when they have used the forum before.
After a lot of testing, trial and error and searching the net I’ve finally got around most of the characterset problems. I’ve written a special script to handle the corner cases which seems to work just fine.
So far it looks like there are only 453 posts and one topic title that won’t be readable after the import (1.17% of the total posts, I’m pretty pleased with that). I’m trying to find a way to repair these posts too. But it looks like we’re at the home straight now.
If you read this, then the forum is moved to the new host.
There are two things that need to be done still:
Search cache update doesn’t work. This is either a bug in the forum or there are still a few weird characters in posts, which brings me to the next item:
Update the posts with a lot of ‘???’ in them. This requires some manual work and I will do this later.
Update the post with double byte encoding problems in them like in this topic title: Abstimmung über Straßen.
After that:
Add language files
Go further with forum adapting to user needs as discussed in previous topics.
Yeah, success finally. It took about 500 lines of PHP code (lots of copy paste though) but the utf8 encoded latin1 characters (or the other way around, I don’t know) have now been converted to their old format and successfully updated in the database. The German and Scandinavian topics shouldn’t have many problems anymore.
There are still a few posts with problems though: There are some posts with quotes that contain both latin1 and utf8 encoding. Such posts can’t be fixed automatically (you’d screw up the utf8 or latin1 part). I guess we will have to accept these. Maybe, maybe (and I mean maybe!) I will fix those by hand some day.
So in the end this forum with mixed utf8, latin1 and html encoding in posts, topics and user definitions has successfully been converted to full UTF8.
There still was a nasty bug where people with non-latin1 characters in their username were not properly handled. And another one where new users could not be created in the db. This problem was discovered yesterday and both are fixed about half an hour ago. The non-latin1 issue was fixed by converting the authentication response from the OSM api to proper UTF8 first. The login issue turned out to be a fieldname in the INSERT query that did not exist anymore in the new database.
I acknowledge the problem. Don’t know where it comes from yet. I didn’t experience it while testing though.
Edit: I think this problem is fixed: the password was stored with md5 in the db while the session cookie had the password in sha1 format, so when the forum tries to review the cookie it sees a different password and as a result you’re logged-out. The login script now checks for the same capabilities as the rest of the forum and will choose the strongest encryption available.
Editing message is completely unaviable. “Error: Unable to insert search index words” with no effect. Adding new message is a bit crasy - it adds same message each time refreshing the page with screen “Error: Unable to insert search index words.” In combination with no possibility to delete own posts it results with multiplying messages. Happens in some last days.
Unfortunately it seems that especially the Cyrillic topics and posts are a mess with regard to the character set in which they used to be stored. This breaks the search function as well. I’ve now disabled the search index update when a new posting new messages, so the error message should be gone now. But the search still doesn’t work like it’s supposed to.
