OSMF should host a Forgejo/GitLab instance

vorpalblade-kaart · 2024-05-02T13:58:43.092Z

I’ve been slowly working on converting JOSM plugins from svn to git.
The big problem there is I’m doing the following:

Attempting to map svn and patch users to git users for attribution purposes
Trying to ensure that we keep svn:externals (this is harder than it seems; I’m having to do a lot of manual work for this)
Split out plugins into their own repos (and add them to the git modules from 2, which is a bit harder than I’d like)

It also doesn’t help that I’m trying to ensure that as much history is kept (so I’m having to look at different repos and going “this repo is a parent of this other repo, so it should be in the history of this repo”).

As far as JOSM core goes, we can do something like git rev-list --count HEAD; we’ll probably have to have some kind of additional increment, if we don’t want to “reset” to an earlier version, but it is doable:

With any luck, I’ll nail the plugin repos down this month, and I can start transferring them over to the JOSM GitHub organization, and then I can poke stoecker to make the repos on JOSM’s server.

Probably not, since it has been more involved than I originally thought. Mostly due to me trying to keep as much (good) history as possible. Random factoid: someone effectively did a “ctrl+s” quite often in the plugin svn repo.

In any case, moving to git would be a blocker for all of the alternatives to what we have. In the best case scenario, we would be unable to move to gitea, GitLab, GitHub, etc. without being on git.

P.S.: The blog post that “no longer exists” is now at pantierra's Diary | The use of Free and Open Source Software in the OpenStreetMap Foundation | OpenStreetMap . Things are fun when people change their username.

pnorman · 2024-05-02T18:54:01.655Z

vorpalblade-kaart:

Attempting to map svn and patch users to git users for attribution purposes

There’s a list from converting OSM SVN - would this help?

contrapunctus · 2024-05-02T19:32:04.836Z

Following up on my offer to set up and maintain a Forgejo instance, I asked on IRC who to contact. @SimonPoole referred me referred to @Firefishy , who (while voicing his disapproval for the idea of hosting a forge) suggested requesting a dev server account, although @TomH pointed out that the dev server is “absolutely not an appropriate place to run a production service”. I was granted a dev server account, with an email from Firefishy stating (I hope it is okay to quote the relevant excerpts) -

[…]

The dev server is not the right location for hosting a Forgejo / GitLab or other similar service. It is for development/testing of OpenStreetMap software.

Self hosting a Forgejo / GitLab instance is not a small task. A personal instance, with 2 or 3 users is very easy to manage, but becomes a lot more difficult with 20+ users, with High Availability, CI, Upgrades/Downtime and Permission/Groups etc.

[…]

We are not sufficiently resourced to take on self hosting Gitlab. I do not see a justifiable benefits for us investing in the amount of work required. […]

So I am, once again, seemingly at a dead end. If anybody has any suggestions, please let me know.

mikeocool · 2024-05-02T20:35:54.312Z

Having operated self hosted VCS and CI/CD systems for several organizations, I very much support @firefishy’s call here – as several people have stated, the OWG’s very limited time and resources are best spent on OSM’s unique needs – that is running OpenStreetMap. If the API, OSM.org, and the associated services aren’t running smoothly, there’s really not a lot of value in having a custom setup VCS and CI/CD for them.

However, I think you have some potential avenues to explore:

Given that OSM projects don’t have particularly unique needs in this realm, perhaps there is an existing more general open source consortium that would be interested in taking on the work building out a more open VCS and CI/CD system (or already has), and you could work with them. If not, presumably there are other people in the wider open source community that have similar concerns – and you could start to build such a consortium to take this on.

Alternatively, though many people don’t leverage this much these days, git is a distributed VCS. Nothing is stopping someone from mirroring various OSM related repos on a self hosted git server. Folks who dont want to use github, but do want to contribute to OSM-related projects could push branches or forks to that git server, and then make requests to the core project maintainers to pull branches from that server into the primary repo (i.e. git pull mirrored-origin some-feature-branch). While it might be an annoying ask of a core contributor, it’s certainly less of an ask than having them fully move their VCS and CI/CD.

This would also be a fairly low stakes of getting started with this, since the server wouldn’t be the primary VCS for these projects, there are many things you wouldn’t need to worry about up front. And if this way of working started to take off, perhaps you’d be able to convince various projects to move their primary VCS to that server.

That said, for either of these, someone would still need to take on the expense the server (s) and the work of setting it up and maintaining it.

ZeLonewolf · 2024-05-02T21:00:40.933Z

contrapunctus:

So I am, once again, seemingly at a dead end. If anybody has any suggestions, please let me know.

GitHub. The suggestion is GitHub.

vorpalblade-kaart · 2024-05-02T21:00:55.809Z

@pnorman: Not really – I think I used the OSM SVN git repo (from GitHub) for some users. Thanks for the offer though.

At present time, I’m missing “good” attribution information for 94 of 9318 commits (~1%). Please note that some of these would not be in the OSM SVN mapping since I attempted to convert most “patch by <foo>” to have authorship by “<foo>” committed by whoever did the actual commit.

Missing emails

I’ll note that some of the names were from a wiki page on the OSM wiki. So they may be incorrect.

Commits	User
1	Binnette Binnette@svn.openstreetmap.org
1	BitSchupser BitSchupser@svn.openstreetmap.org
1	Galo Higueras galo@svn.openstreetmap.org
1	Gnonthgol Gnonthgol@svn.openstreetmap.org
1	Jason Huntley Jason_Huntley@svn.openstreetmap.org
1	Locked Locked@svn.openstreetmap.org
1	MonkZ monkz@svn.openstreetmap.org
1	Morten Olsen lysgaard@svn.openstreetmap.org
1	RafalR RafalR@svn.openstreetmap.org
1	Robert robert@svn.openstreetmap.org
1	Robert Schedel robert_schedel@svn.openstreetmap.org
1	TAA TAA@svn.openstreetmap.org
1	anonymous anonymous@svn.openstreetmap.org
1	delta_foxtrot delta_foxtrot@svn.openstreetmap.org
1	francois2 francois2@svn.openstreetmap.org
1	george-hopkins george-hopkins@svn.openstreetmap.org
1	istepan istepan@svn.openstreetmap.org
1	javiersanp javiersanp@svn.openstreetmap.org
1	julianladisch julianladisch@svn.openstreetmap.org
1	kre3d kre3d@svn.openstreetmap.org
1	ksmlgl ksmlgl@svn.openstreetmap.org
1	langoor langoor@svn.openstreetmap.org
1	mark.reidel mark.reidel@svn.openstreetmap.org
1	p.janaszek p.janaszek@svn.openstreetmap.org
1	skorbut skorbut@svn.openstreetmap.org
1	svenmeier svenmeier@svn.openstreetmap.org
2	André vsandre@svn.openstreetmap.org
2	Chris Tipper Chris_Tipper@svn.openstreetmap.org
2	Erik Gruschka erigrus@svn.openstreetmap.org
2	Fabian Kowitz fkowitz@svn.openstreetmap.org
2	Fran Prieto franpd@svn.openstreetmap.org
2	Hassan Sabirin hassans@svn.openstreetmap.org
2	Jörg Possin joergp@svn.openstreetmap.org
2	Preferred Preferred@svn.openstreetmap.org
2	abhilekhsingh041992 abhilekhsingh041992@svn.openstreetmap.org
2	ax ax@svn.openstreetmap.org
2	r_x r_x@svn.openstreetmap.org
2	s8evq s8evq@svn.openstreetmap.org
3	Chris Vodry vodryc@svn.openstreetmap.org
3	Nikhil Shirahatti snikhil@svn.openstreetmap.org
3	Rafał Jachowicz rjachow@svn.openstreetmap.org
3	openbrian openbrian@svn.openstreetmap.org
4	Jean-Marie Dubosc jmdubosc@svn.openstreetmap.org
5	Clement Menier clementm@svn.openstreetmap.org
6	Jan Petranek casualwalker@svn.openstreetmap.org
6	Julio Rivera julior@svn.openstreetmap.org
11	Martin Petricek bilbo@svn.openstreetmap.org

Most of the rest of the attribution information I managed to mine from various public sources (I preferred contemporary sources over more recent ones).

I’m not going to claim that the attribution is perfect, but it should be pretty good.

contrapunctus · 2024-05-02T21:49:20.378Z

@RichardF - On one hand, my exhortation to keep the discussion constructive is deemed “insulting”…on the other hand, I haven’t seen any moderator say anything about @ZeLonewolf 's dismissiveness towards concerns (on the mailing list) nor this low-effort trolling.

contrapunctus · 2024-05-02T21:57:37.029Z

Codeberg.org, mentioned in my first post, is exactly what you describe. But people are reluctant to move there, too. So I hoped an OSMF-run forge would have more trust and the OSM SSO would remove the need to create new accounts.

Re: mirroring - as mentioned earlier, that’s my nuclear option. If upstream developers aren’t interested in moving off GitHub, I can fork repositories to Codeberg (including issues, PRs, wiki, etc) and contribute there. I doubt they’ll care enough to merge changes, which means we have an easily-prevented split in efforts. The repositories themselves could technically be kept in sync, but the non-Git data will drift (there’s no synchronization mechanism for that, AFAIK).

ZeLonewolf · 2024-05-02T21:58:50.177Z

It seems pretty clear that OSMF doesn’t have the appetite to do this (because github exists), and you’ve managed to step off on the wrong foot by insulting people that disagree with you. And there doesn’t seem to be anyone willing to donate servers for this project.

Sooooo it’s starting to sound like Sisyphus pushing the rock up the hill, and I would agree with your assessment that you’re at an impasse.

Minh_Nguyen · 2024-05-02T23:35:42.741Z

contrapunctus:

Re: mirroring - as mentioned earlier, that’s my nuclear option. If upstream developers aren’t interested in moving off GitHub, I can fork repositories to Codeberg (including issues, PRs, wiki, etc) and contribute there. I doubt they’ll care enough to merge changes, which means we have an easily-prevented split in efforts. The repositories themselves could technically be kept in sync, but the non-Git data will drift (there’s no synchronization mechanism for that, AFAIK).

Folks around here are quite familiar with this scenario, actually:

The Next Generation of OpenStreetMap — in Python! General talk

Greetings, OSM Community Today marks the unveiling of my most ambitious project yet: OpenStreetMap-NG. Nearly a month in the making, I am super excited to bring this to light. This project is a transformative rewrite of the foundational OSM infrastructure1,2 to Python. Its goal? To simplify contributions, secure against vandalism, and modernize the API, all while remaining backwards compatible. (Some) Highlight New Features 100% Python; Move away from the complexities of Ruby, …

Unfortunately, as you might’ve guessed by now: GitHub.

NorthCrab · 2024-05-03T02:20:00.830Z

I am personally all for free git alternatives. Unfortunately, today’s reality is that GitHub has the widest contributor audience. You can compare how well different projects do across different platforms. On GitHub, you will always find more reported bug reports and more contributions - which really matters for this kind of project. For private/small projects, it makes sense to host them on alternatives. For more substantial projects, and especially those that are made by the community - GitHub is the only sensible choice today.

contrapunctus · 2024-05-03T05:06:17.685Z

…and Google Maps is “the only sensible choice today” in India. The POI, bus route, and 3D building data data is vastly more complete and detailed, and there’s real-time traffic data, PT timetables, etc. Yet, us free software advocates avoid it and spend countless hours contributing to OSM, organizing mapping parties, designing promotional material, and conducting advocacy at conferences instead. Should we have done the same in this case, and contributed to Google Maps instead?

There are times when we have to do better than swallowing the bait used by proprietary platforms, and look at the bigger picture…no matter the network effect and no matter how much the world screams at us to do the opposite.

A large project like OSM moving to a free platform would in fact be a big boost for free platforms. It would bring more contributors to these forges, and maybe even inspire other projects to consider following suit.

NorthCrab:

You can compare how well different projects do across different platforms.

If others are also applying your reasoning, and thus only small projects are moving to other hosts…they would naturally not receive too many issues and PRs on either platform. Does your comparison account for that?

I also don’t see anyone questioning the popular wisdom of “if it’s not Github, there will be fewer contributors”. It is my strong suspicion that the platform doesn’t affect contributions at all - people contribute to OSM software not because they found it on GitHub, but because they are invested in OSM in some way other the other. They would contribute no matter what platform it’s on.

SimonPoole · 2024-05-03T08:27:27.484Z

While I agree that the dev machine is not ideal for non-experimental projects, on the other hand there is a long tradition of services being run on it that are considered essential by large parts of the community. At the same time there is no concept from the ops side of things of moving such services to a different, permanent, home, or even taking over such services, so I guess it is what it is.

Further we have a volunteer that has offered to do the work to support the service, so I don’t quite see what the workload issue is supposed to be (at least not different than for anything else running there). I guess it would be nice if the OSMF provided a location to store backups, but that can be worked around too.

Back to the topic at hand: there are plenty of OSM-specific projects that are not really looking for external developer contributions, take for example the editor layer index, NSI and so on. These could easily (modulo CI) be hosted on such a service without loss of potential contributors, it could actually open things up the other way around.

The 2nd realistic use case (as I already pointed out) is to provide a secondary issue tracker for projects (likely in conjunction with a mirror of the github/gitlab repo) that can be accessed with the users OSM login. Obviously there is no guarantee that any projects will take up such an offer, but it would seem really low hanging fruit to provide.

Minh_Nguyen · 2024-05-03T15:01:14.082Z

SimonPoole:

The 2nd realistic use case (as I already pointed out) is to provide a secondary issue tracker for projects (likely in conjunction with a mirror of the github/gitlab repo) that can be accessed with the users OSM login. Obviously there is no guarantee that any projects will take up such an offer, but it would seem really low hanging fruit to provide.

I wonder if this could just be a forum category that has the Solution button enabled and maybe auto-closing threads after some period of inactivity. I’ve participated in some less tech-focused open source projects that already use Discourse as a secondary feedback channel. For example, Gramps has an intake category so that their users, who are mostly genealogists, don’t have to know their way around the project’s Mantis bug tracker. Mantis remains their primary bug tracker, but it’s up to the developers to curate it based on the most actionable feedback coming through Discourse.

I think this is already how some OSM projects are using chat platforms like Slack. If projects desire something closer to a chat system, Discourse has that capability too.

SomeoneElse · 2024-05-03T15:29:44.134Z

contrapunctus:

So I am, once again, seemingly at a dead end. If anybody has any suggestions, please let me know

Not really - you’re just waiting for some person or group of people who shares your view about Github with available time, energy and cash to consider sponsoring a server. I’ve no experience of hosting gitlab et al, but based on this and setting up similar “small one job” servers I’d expect the actual server requirement to be small - based on e.g. this maybe less than €10 per month. I’d expect the bigger committment by far to be keeping the server updated, and keeping on top of user requests and occasional misuse.

As I said back here there’s no reason this needs to be an OSMF badged service on OSMF hardware (at least not initially) as it’s a proof of concept. If there really is a demand for non-Github hosting, it may well be a success. If there isn’t, it probably won’t be.

SimonPoole · 2024-05-04T07:41:14.099Z

Minh_Nguyen:

I wonder if this could just be a forum category t

At least in my case users are free to open a topic on the forum and in the past have done that many times and they will get an answer if appropriate, however that is not a replacement for working through an user detected/experienced issue in a structured fashion. Some projects use github discussions for the unstructured bit BTW.

IMHO such a process is far too noisy for a general purpose forum and far too susceptible to noise from otherwise uninvolved bystanders. Not to mention other issues like closed threads remaining visible.

SomeoneElse · 2024-05-04T08:28:38.575Z

SimonPoole:

Minh_Nguyen:

I wonder if this could just be a forum category t

At least in my case users are free to open a topic on the forum and in the past have done that many times and they will get an answer if appropriate

(we’re veering away from Github vs alternatives but)

It doesn’t need to be a separate category - you can use tags very much like categories - you’ll notice that that list has a “new topic” button. I’ve often directed DWG correspondents that way when they want to talk about a country that doesn’t have a category.

From the other side of things, many of the OSM-related software repositories (the website, mod_tile, etc.) get “issues” raised that aren’t software issues at all, but instead “I was trying to do something with OSM data and something went wrong - help!” - those would be far better on this forum than in Github (or any equivalent).

Minh_Nguyen · 2024-05-04T16:56:21.648Z

(Yes, I know I’m gently steering us away from window shopping for Git frontends. That’s called “taking a step back”.)

SimonPoole:

At least in my case users are free to open a topic on the forum and in the past have done that many times and they will get an answer if appropriate, however that is not a replacement for working through an user detected/experienced issue in a structured fashion. Some projects use github discussions for the unstructured bit BTW.

Right, Discourse isn’t a full replacement for a bug tracker. It’s more of a solution to the problem of people relying on proprietary chat platforms than the problem of relying on a proprietary platform for software development. But many of the arguments seem to apply equally, particularly the concern you expressed about sharing personal information or having to maintain an account on an unfamiliar service. (Discourse’s UI is also much more comprehensively localized for a global audience than GitHub’s, which isn’t saying much.)

Some projects do use GitHub discussions, but that feature tries to replicate Stack Overflow post voting just like this forum’s post voting plugin, with all the attendant confusion. I wouldn’t recommend it.

lonvia · 2024-05-05T09:10:30.517Z

I think this discussion somewhat mixes reporting problems and tracking issues. Those two things may look like they are one and the same on the surface, but this is a misconception. They are different problems and need different tools.

Most users who report problems do not have the experience to use an issue tracker in a structured way that helps developers with tracking. It is not rare that the first handful of comments is spent with a back and forth trying to understand what the actual problem may be. This is a necessary part of problem reporting but really not helpful for structured issue tracking.

Personally, I somewhat prefer when people start a discussion here in the forum when they see a problem with Nominatim. It’s prefect for asking clarifying questions. There are lots of knowledgeable people here, who can help with background information on OSM or help tracking down the OSM data that causes the problem. And there are regulars here who are happy to repeat answers to the questions that get asked again and again. I can’t emphasize enough how helpful this is for me as a maintainer. So, when it comes to reporting problems, this forum is just a fine tool for it and it ticks all the boxes on privacy (as far as a public venue can do that).

Issue tracking is a different beast. It has to work for the maintainers and developers. Github currently does the job, not perfectly but well enough. Access to the CI and the ability to communicate across project boundaries are a huge win. Most of the privacy concerns don’t really apply because I do open source development only. Everything is in the open for everyone to see: code, issues, comments. That is a conscious choice. There is also no real lock-in. If you are worried that some information only lives on Github, then there is always the option to mirror/backup the important stuff.

I’m not naive. I understand that there is no such thing as a free lunch, also not on Github. What Microsoft gets out of it, are millions of lines of code they can use without paying license fees. That likely pays for itself a hundred times over. So the deal is: I get an issue tracker and CI, they get the code. I’m fine with that. It’s open source.

InsertUser · 2024-05-05T11:36:40.884Z

lonvia:

I’m not naive. I understand that there is no such thing as a free lunch, also not on Github. What Microsoft gets out of it, are millions of lines of code they can use without paying license fees.

From tech press coverage It seems they also use it heavily for their internal code repositories and some of their early improvements were driven by needs they had managing their own code and teams. They’re also said to have gone moved an “inner source” methodology for all their code which a GitHub variant would be convenient for. IMHO their acquisition was probably more about preserving tooling that they find useful than snaffling code they could already just fork as an when they needed it.

I really don’t think OSM will benefit much from shifting away and it’ll be costly in terms of manpower to run the thing unless we just hand everything over to yet another faceless entity liable to being taken over.