The Next Generation of OpenStreetMap — in Python!

General talk
, , ,
41

I should probably clarify that they were intimidated not by Ruby on Rails or Ruby off the Rails, but rather by a perception that the software is monolithic. I think many people these days expect Web applications to be structured as modular microservices. That is the case with the Rails port, if you consider things like iD and leaflet-routing-machine to be modules. But a common desired use case is to collaborate on a map without features like GPS trace management, diary posts, and image exports.

It’s the breadth of this project that dissuades more casual folks from wanting to maintain their own instance. Maybe that’s fair – maybe openstreetmap-website needs to focus on serving OSM and similarly ambitious community projects. My takeaway from these conversations is that there also needs to be an answer for the long tail of smaller projects, just as in all fairness we couldn’t admonish people for mapping historical features until OpenHistoricalMap became viable.

From this perspective, a true reimagining of the Rails port would consider the broader landscape of what people want to do with geodata, come up with a solution that’s a joy to use, and all the while look for opportunities to better serve an OSM-like project within that framework. That’s a much bigger project than a more literal port of the Rails port’s core functionality with extra bells and whistles, but at the same time, there are fewer obstacles to finding users and getting deployed in some form. And – imagine this – it could be a collaboration among complementary projects rather than something competitive or adversarial.

6 Likes
42

Which could be solved by splitting up the existing codebase rather than rewriting it in another language (and so far this specific port is moving towards being just as monolithic).

As I’ve mentioned previously I think there is real value in doing so, even more than an initial change of language. As you say that could open the individual components up for more users and contributors who don’t want the full stack, and it would also open the door to replace some parts of it with other languages och technologies (for example, someone in this thread mentioned a high performing Rust port of the API).

The big downside with a more split up codebase however is that the total maintenance cost tends to go up for those who do actually care about the entire thing, i.e. the current maintainers och contributors to the current monolithic codebase.

5 Likes
43

IMHO there is literally nothing in that for OSM. Extending the target audience comes with its own costs, not just financial (and FWIW we literally already don’t have money to fund current operations), and it isn’t as if we’ve run out of OSM contributor focused improvements to implement (far from it).

We already have exactly this scope problem in the current project with the bits of the API that are duplicated in cgimap, OSM doesn’t need the Rails versions, they are just there for use outside of OSM.

PS: at the time I closely followed Hectate closely as it was clear that it was originally intended to pull the rug out from underneath OSM. I don’t think it needs yet another unsuccessful sequel.

3 Likes
44

I think this thread is mostly proving that there are as many opinions about preferred development environment as there are programmers.

At the risk of adding another one, I am bemused at the focus on replacing the backend. The backend is fine. It’s basically a CRUD app and as such the benefit of rewriting in Python or Rust or Zig or Brainfuck is only ever going to be marginal. The opportunities with osm.org are chiefly on the frontend. If I were to dedicate days to improving osm.org, I’d focus on replacing Leaflet with Maplibre rather than Ruby with Python.

15 Likes
45

I didn’t say anything about OSMF or funding. The context of this discussion, lest we forget, is that someone has put forward a rather significant amount of code without support of any kind from OSMF. If that kind of effort can be channeled toward complementing core OSM software instead of competing against it… what’s the concern again?

1 Like
46

But you did suggest rampant scope creep :slight_smile:

47

You’re right, I did. The core OSM software is FOSS, so people are generally allowed to scope-creep unofficially. If the project in question had been framed as something other than the “next generation” of OSM, we might not be having this discussion but might instead be cheering it on as a sort of on-ramp to the rest of the OSM software ecosystem. Instead of starting a language debate, it might have inspired other efforts for other languages. To each their own.

3 Likes
48

I still like COBOL. So straightforward.

1 Like
49

Monolith = bad, microservices = good certainly has been a commonly repeated story in the last decade of web development. It seems this hype train is finally losing steam though, with developers finally realizing that the overhead is often not worth it.

6 Likes
50

Yes! The core issue I am trying to address is project stagnation due to its monolithic design. Implementing version 0.7 alongside 0.6 in the current Ruby website would require an immense amount of work, while NG is designed for multiple API versions from the ground up. I am also mostly proficient with Python. So, it makes more sense to work in Python and make this project the best it can be! This will also open up more project possibilities, as there are approximately 8 times more Python developers than Ruby developers, leading to more contributors and a more community-driven project! Imagine (theoretically) having 32+ active contributors compared to just 4.

I do agree that frontend also requires some work, and by making the backend less monolithic, this will finally make it realistic to achieve. I want to focus on the core issues and those of the utmost importance. By modernizing the backend, not only will frontend updates require less work, but many issues with 0.6 and vandalism will (hopefully) be resolved. It seems logical to prepare the ground first and then start working on the dependent code.

I know long threads may be hard to follow for the general community audience. I will post an Update 1 video approximately on November 16th and no later than November 23rd - discussing current progress, general discussions, etc., all condensed into one simple video.

I will resume very active development of NG on November 14th and no later than November 21st. I have a project due that I am currently working on, and this was originally accounted for in the Roadmap, so nothing changes there.

Thank you all for the feedback and such an amazing open discussion :slight_smile:!

8 Likes
51

Previously, it was suggested to discuss first, and then make a decision. Also, I want to remind that for a long time there has been a discussion about making changes to the data scheme, there was even a study commissioned by the OSMF.

Changing the data schema will also require changes to the API, so that should also be considered in the future development. A discussion of data schema changes takes place here (osmlab/osm-data-model · Discussions · GitHub). If you have suggestions, your own take on data organization, leave your feedback there, or create a new discussion on this forum.

I hope that @NorthCrab’s work will lay the necessary foundation for the next implementation of changes to the data schema.

4 Likes
52

As the saying goes, People Who Say It Cannot Be Done Should Not Interrupt Those Who Are Doing It, eh?

We already have exactly this scope problem in the current project with the bits of the API that are duplicated in cgimap, OSM doesn’t need the Rails versions, they are just there for use outside of OSM.

So what is the issue you’re having really with @NorthCrab attempt?

Either @NorthCrab will manage to pull it off, or he won’t. It’s not out of yours (or mine) pocket. I also have serious doubts that it is doable and would certainly not bet my own money on success (mainly due to politics in OSM, and not his technical ability or technical superiority of new solution!), but the guy seems all-in on it and full of youthful enthusiasm, so I wish him all the luck. It’s his engineer-months on the line, not mine.

I say let him do it and then see see how good/bad the results is. It is like being given free box of lottery tickets – why not take it, even if you know the chances of winning are very low? It’s free!

For the record, I certainly much prefer python to ruby on rails (although I don’t really program in either, but have tried to dabble in both, with hugely more success in python attempts). I hugely appreciate the herculean effort that RoR maintainers are putting on that code base, but if we are expecting new contributors to jump in, ruby on rails is IMHO bad choice. But hey, I’ll at least admit the time has run down that chance to rewrite OSM in perl (which is my preferred go-to language)…

Just like I don’t REALLY believe Elon Musk will succeed on putting people son Mars this decade (much less by 2026), but I sure applaud the dream and enthusiasm and effort and money and [if you insist] pure crazieness pumped into it by people who do believe in it, and DO like to watch those gargantuan rockets (attempting to) take off, whichever their fate may turn out to be. Excitement guaranteed! At least it is attempting to move somewhere in human lifetime, even if it won’t work. If I were to place my bet on NASA or ESA for that dream, you can guess why the enthusiasm would be lacking… s/NASA/Ruby on rails/g; s/SpaceX/python/g; :smiley_cat:

13 Likes
53

Oh no, it seems that I’m late again and didn’t see it until this thread was so active. I hope this comment will not cause disturb to other participants.

First of all, as a developer who mainly uses Python, it’s nice that someone can make OSM become like a python (just kidding (that’s another joke, why are you calling a crab)) It really makes it easy to attract developers to participate because python truly used more widely.

However, please note that for a project of such a large scale as OSM, any disruptive change is a disaster. Even if you reimplement the backend of the OSM API in any new language, you must ensure that it can be compared with the current one. projects are compatible, like LEGO bricks. This means that you’d better implement API0.6 first (in fact, my own python reading and writing library is also considering implementing support for multiple versions of the API (including the one that Jochen Topf is researching) by overloading classes, but the most important thing is It must be the one currently being used)

Sorry, I’ve seen your reply on GitHub, thank you for making stable transition an important part of the new backend.

(There may be some overly excited comments about MongoDB here, which have been deleted.)

After reading the reply on GitHub, I found that PgSQL support will still be considered, so I feel relieved. Please note that this does not mean that I expect to prevent all change

You can rest assured that OSM has a lot of room for improvement, both from the front end and the back end. The core bottleneck may be the lack of a reviewer with enough time and a person who has the time to write code and contribute.

Additionally, this is a good way for us to add more useful features to OSM, such as support and search of hashtags that sparked a very heated discussion in another thread

Finally, if you choose to reimplement an ancient backend in a modern language in the last few months of 2023, then why not consider trying Poetry, the most modern Python package manager currently available?

4 Likes
54

Thanks! I will consider it later on. Typically, I just use pipenv because it just works, but I will definitely check out Poetry :slight_smile:.

1 Like
55

Because nobody seems to have asked it till now, I am going to.

What would be the test to say that any implementation faithfully implements the OSM API version 6? Does this already exist, if not, shouldn’t this be prioritised? I believe this would be essential for a better onboarding experience of new developers to the existing rails port as well.

If there were an agreement on replacing the rails port with something else or just doing a major overhaul on the existing rails port( to maybe modularize it better? ), is there a plan for what such a production migration would look like? Maybe some sort of production parallel to replay traffic on the new implementation to check that the state matches with the existing implementation at the end of the replay. This sort of thing would be needed to give more confidence that any major changes won’t break things in case a comprehensive api test is not possible.

Probably people who have enough time can pitch in for making the above two things happen, as they are clearly things that are good to have, both to maintain the current codebase and any future versions.

And also coming from the ops perspective, who currently maintains and monitors the rails port in production? Is it the ops team? If so, how comfortable is the ops team with maintaining applications which serve live traffic in python… because they do come with their own set of headaches and corresponding diagnostic tooling.

10 Likes
56

Not directly answering your question but since (at least JOSM) lets the user enter API endpoints it’s rather obvious to pull up something like test.api.osm.org, point JOSM there and try doing various things, and see whether it breaks.

Apart from obvious automated API tests.

3 Likes
57

I have a general idea of how it should be approached, but this is of low priority for me at the moment. I first focus on development, and only later will I prepare the necessary migration tooling. I believe it’s most optimal to have a stable code base first.

This is one of the ideas; both apps could process the same traffic, and then a planet diff could be compared to check for potential errors.

The application will run in a Docker container, which is a standard setup in the OSM infrastructure. Any external integration, for example with Prometheus, will be done as part of the NG project. Integrating this project will not require immense amounts of work; it will involve a basic setup procedure.

58

Hello Community, due to some unaccounted for events (personal & other projects) and a flu, NG development was on hold for a week. I will resume active development in 1-3 days and will adjust the roadmap ETAs by 7-10 days to account for this delay.

9 Likes
59

The nature of volunteer open source projects is that they get there when they get there based on individual volunteer time and interest. So I wouldn’t be too worried about timelines and roadmaps – as a fellow open source maintainer, just update us when you make breakthroughs, accomplishments, or have demonstrations to share! It’s not a race.

21 Likes
60

I think this is the most important element of the platform/language x vs y for a FOSS project discussion: if x has a shrinking pool of developer talent, and there’s no migration path, that might not not bode as well for future as alternative paths.

Change management practices, monolithic nature, etc., are frictions - possibly small, possibly unwarranted, but still real frictions - that further reduce that shrinking contributor talent pool.

I agree with Minh’s comment above that the name “NG” is a bit off-putting to those already in the community, but imagine if you were new - would you want to work on something called “OHM - Maybe Future Alternative”?

My experience with OpenHistoricalMap and associated difficulties of finding and staffing Rails- and devops-focused talent, makes me wish there were 2 or 3 projects like OHM-NG, so there might be a bit of competition on this front.

Anyone who can spend time on improving the current platform and seeing how it works, that would be awesome. But, they might have an even bigger impact on an alternative, esp if they are the 2nd or 3rd dev. So, I’d say - if Python is your jam, jump on OHM-NG and start coding.

In the meantime, Rails is the world OHM is living in & I expect that it will be for the next several years at least. In parallel, I’ll be cheering on OHM-NG with the challenge, “It can be yours, but if you want to be the NG, earn it!”

2 Likes