Can you create a static version of the old site?

I cannot share the database dump. It contains private data (password hashes, messages, etc).

I do not want another framework to keep the database and site alive. I am strongly in favour of pure static HTML archive.

I am happy to run any created scraping code to create the static input data.


import requests
from bs4 import BeautifulSoup
import sys
import json

start_page = 1
end_page = 88268

url_template = "{page}"

data = []

for page in range(start_page, end_page + 1):
    url = url_template.format(page=page)
    response = requests.get(url)
    soup = BeautifulSoup(response.content, 'html.parser')
    element = soup.find(id="CALeft")

with open('qnadump.json', 'w') as f:
    json.dump(data, f)

Then, copy qnadump.json, style.css, and into the “questions” subfolder on htdocs. :

import json
import sys
import os

with open('qnadump.json') as f:
    data = json.load(f)

for i in data:
    nya = '<meta charset="UTF-8"><link rel="stylesheet" type="text/css" href="../style.css">' + i['text']
    os.makedirs(str(i['id']), exist_ok=True)
    file_path = os.path.join(str(i['id']), 'index.html')
    with open(file_path, 'w', encoding='utf-8') as file:

Run there. It will generate a lot of folders, like this one (although I stopped the process after creating only two for this example below) :


Here’s the final result :

Note :

  1. TODO : Since it will be converted into a static version, search and tag functionalities will be quite challenging to reimplement.
  2. Fortunately, the user’s avatar is stored externally on Gravatar, so we don’t have to manually scrape them one by one.

@rtnf would a shell account on our dev server help you? I use my account on that server to run a few python scripts too.

Thank you for pointing that out.

By using that shell account, I can create a static site clone. But where should I submit that static site so it can be deployed properly? (probably replacing the old Django-based with a static site)

Don’t worry about deploying the replacement site. I will handle that once a static copy has been created.

1 Like

Ok. I have submitted the account request.

Starting the scraping process right now…



Yes, I can

You likely want to run the scrape via a background session using GNU Screen or tmux.

I use nohup (no hang out) to run the scraper in the background

I got spare time. I’m happy do a static copy by scraping.

LMK if there’s someone else working on it or if I should give it a go.

Can I cash in my reputation points? :slight_smile:


Update : 100% scraped. It’s time to reconstruct everything into a static web :


I want to upload the scraped data to Github (so I can download it, and process it locally), but apparently it’s way too large.


So, I have to process everything on the server.

And it’s now done… (just play around with the thread ID there)

Additional notes :

A. Not every thread ID is accessible, since some of them are already deleted from the original

B. Low quality answers are not scrapped. Only answers shown on first page are scrapped.

C. Some of the files are still hosted on (and not scrapped). Make sure that all of this files are not deleted when turning off the OSQA django instance.

1 Like

Update : Homepage and (title-based) search functionality now already reimplemented :

1 Like

Feel free to have a go. May the best implementation win.

You could use something like this to “compress” the json. The format remains valid, just whitespace is removed.

jq -c . < input.json >output.json

Wasn’t all of this going to be imported to this forum at one point?

1 Like

I asked the same question before and here’s what I got back:


@rtnf How you getting on creating a static clone of ?

Please try ensure the functionality and URLs do not change as far as remotely possible.

Something looking and working very similarly to is the ultimate goal.

Note it is also possible to host web content directly under your user on the dev server. eg: serves the contents of /home/rtnf/public_html/ directory.

I noticed that both the user profile and tagging functionalities are disabled in the archived version of the Wireshark forum (reimplementing both features using static HTML is quite a challenging task, and they seem to be giving up altogether on this.).

The last problem in my implementation is likely to be the URL change, which could probably be fixed by using Apache’s mod_rewrite (for example, automatically redirecting “/questions/88266/osm-carto-multilingual-tags” to “/questions/88266”). Additionally, I need to (1) add a link to the homepage on each answer page, allowing users to navigate back, and (2) make visual improvements to the homepage.

I’ll try to do it soon & post some updates here.

Update 1 : Homepage

Update 2 : add a link to the homepage on each answer page (click the osm logo to go back)

Update 3 : All of the old URL can now be preserved (by automatically redirecting “/questions/88266/osm-carto-multilingual-tags” url format to “/questions/88266”) (this will be automatically redirected to

.htaccess ModRewrite config :

RewriteEngine On
RewriteBase /	
RewriteRule ^questions/([0-9]+)/[^/]+$ questions/$1 [R=301,L]

If we aren’t going to archive user pages is there going to be an email to all users that they have XXX days to go there and manually archive their contributions?

I had a go at this for mine before the original shutdown date and a really janky version of it kinda sorta works with httrack (would be way easier if the number of results per page was increased significantly).