Nix package manager discussion

Woazboat · 2024-03-20T18:45:23.780Z

The Next Generation of OpenStreetMap — in Python!

The only hard dependency is the Nix package manager

That is a pretty steep hard dependency.

Last I checked there weren’t any installation instructions for openstreetmap-ng, has that changed?

the PostgreSQL database runs under the current user

Running different parts of the service as different users with limited permissions is a feature, not a bug.

(As a sidenote, looking at the Nix website: Who the hell thought recommending piping curl output into sh as the default installation method was a good idea)

whb · 2024-03-20T20:51:06.003Z

Woazboat:

As a sidenote, looking at the Nix website: Who the hell thought recommending piping curl output into sh as the default installation method was a good idea

Rust?
https://www.rust-lang.org/tools/install

NorthCrab · 2024-03-20T21:12:45.547Z

Woazboat:

Who the hell thought recommending piping curl output into sh as the default installation method was a good idea

Could you please elaborate on that? How in your opinion running a shellscript differs from running a binary or an installation script, wrapped by some other package manager? When making statements like that, providing some justification would be welcome. I personally enjoy shellscripts because they don’t compromise on security (compared to running binaries), but give you the benefit of being more transparent.

Woazboat:

That is a pretty steep hard dependency.

Also could you please justify this one too? Thank you!

Woazboat:

Running different parts of the service as different users with limited permissions is a feature, not a bug.

If you do that when performing local development, you will need sudo whenever you want cleanup after the database - which is annoying and dangerous. By running the database as the same user, you won’t need sudo permissions. Is there a single benefit of running a database as a different user during development?

Minh_Nguyen · 2024-03-20T21:55:39.338Z

The Next Generation of OpenStreetMap — in Python!

Woazboat:

Who the hell thought recommending piping curl output into sh as the default installation method was a good idea

Could you please elaborate on that? How in your opinion running a shellscript differs from running a binary or an installation script, wrapped by some other package manager?

There was that one time everyone who used codecov.io sent their AWS secrets etc. to an unknown third party because they followed codecov.io’s instructions to pipe cURL into Bash in their continuous integration workflow and someone snuck a back door into the endpoint. It was also vulnerable to a man-in-the-middle attack even before that. Pipes are kind of magical!

NorthCrab · 2024-03-20T22:09:22.737Z

Yes but you could make the same case for other means of distribution. It’s not specific to the shellscript installs. You could have a backdoor in a binary too, frankly, it would take even more time to discover it . Shellscripts are very transparent.

I wouldn’t want to continue this talk furthermore mainly because it’s quite off-topic. We are discussing something that affects a broad Linux ecosystem, and not this project specifically. If you are security cautious, feel free to run the shellscript line by line (or even build Nix from source), actually it’s quite straightforward and easy to understand. Nobody’s forcing you to pipe it to shell without examining it - but most people don’t care(have enough time), and instructions are generally made for most people.

I’ll leave you with this piece of lecture (just 3 pages long!), talking how it’s impossible to trust any software nor any hardware. It’s a good starting point for further reflection.

There’s also this good video talking about how you are most likely running with multiple backdoors today.

IanH · 2024-03-21T04:40:08.660Z

NorthCrab:

I’ll leave you with this piece of lecture (just 3 pages long!), talking how it’s impossible to trust any software nor any hardware. It’s a good starting point for further reflection

The problem is that you are trusting the whole setup to one application. At first blush, Nix doesn’t even try to use existing platform specific solution for it’s own installation. I would expect it to brew for MacOs or Chocolatly for Windows. Even pip run curl though itself to minimize the chance of a vulnerability. I guess I’m not even sure need for an installer, when a bunch of shell scripts would suffice.

Much of the setup could have been automated though the use of containers such as docker or similar. There are enough tutorials on how to setup a secure database using single config file and a standard image. It feels like you spent a lot of effort to make it easy for everyone.

In general it seems like need to go it alone has lead you to spending a lot of time working on the installation. Instead just creating a single set of solid setup scripts for a for your own environment. Letting others spend time optimizing it for a wider audience.

Hashmush · 2024-03-21T18:08:53.966Z

I, for one, approve of using Nix. For the people that haven’t tried it, it might be a steep dependency, but it simplifies maintenance in so many ways. And, if done correctly, lowers the barrier of entry in a way no other tooling does.

Though I am surprised to see a single shell.nix file in there and no flake.nix.

flohoff · 2024-03-21T18:49:48.150Z

Hashmush:

I, for one, approve of using Nix. For the people that haven’t tried it, it might be a steep dependency, but it simplifies maintenance in so many ways. And, if done correctly, lowers the barrier of entry in a way no other tooling does.

With 35 years of beeing a Software Developing, Network Designer, Linux only Guy, Debian Developer, Sysadmin i say - Nope.

Its about fitting into your operational infrastructure. And dragging in another complete OS with complete new set of operating updates, security implications etc just for the sake of “we want a new hipster tool” its a security desaster just because you add a lot of new complexity.

Flo

citrula · 2024-03-21T18:54:47.150Z

Why was this split into a different thread? It’s directly related to NorthCrab’s project.

NorthCrab · 2024-03-21T19:09:31.779Z

Yes that, I think flake is a great concept but for nix-new people it may be a little too much at the beginning. I like to keep things simple. The shell.nix already locks onto specific versions by not importing pkgs from the system. I think it’s a great balance on usability and ease of use. Also, setting up Flakes requires some additional steps which I would love to avoid. I want to make the development process as straightforward as it gets

NorthCrab · 2024-03-21T19:31:57.444Z

flohoff:

Nope.

I encourage you to give Nix a try. I think the most obvious benefits to any developer, would be reproducible environments (no longer “but it works on my machine”), simplified instructions when getting started into projects (usually just 1 nix-shell command), and sanity (it prevents you from having hidden dependencies). Oh! It also lets you lock onto specific builds of dependencies, it’s like a deps lock file but for system packages .

flohoff:

just because you add a lot of new complexity.

Let me change your mind. You can browse the derivations source of any application, with clearly defined ins and outs. Everything in Nix is deterministic. For example this htop derivation to not get you overwhelmed. Even in 30 years from now on, the same derivation tree will build and run in exactly the same way - isn’t reproducibility great? I have never seen it as any more complex compared to other package managers. I would say, that by having clearly defined dependencies you avoid lots of ambiguity, making it simpler to reason about.

If you still disagree, can you provide some more context or examples? I would perhaps be able to address some specific issues (instead of having my replies so broad).

starsep · 2024-03-21T20:23:38.794Z

Why was this split into a different thread? It’s directly related to NorthCrab’s project.

I asked for it. It’s related but topic went off-topic, especially with curl | bash discussion.

Its about fitting into your operational infrastructure. And dragging in another complete OS with complete new set of operating updates, security implications etc just for the sake of “we want a new hipster tool” its a security desaster just because you add a lot of new complexity.

Note that Nix doesn’t require using NixOS. It can be run on e.g. Linux.
I undestand “we want a new hipster tool” as “running shiny new thing”.
Nix is getting popular lately but it has 20 years.

@NorthCrab Do I understand correctly that it’s only for developer setup, not production deployment?

If yes then it’s really alternative to installing dependencies manually in the system or using containers (e.g. Docker).

I think Nix is a sensible choice. I am personally using it as package manager. It’s not trivial to start but reproducibility is great.

NorthCrab · 2024-03-22T00:29:16.944Z

starsep:

Do I understand correctly that it’s only for developer setup, not production deployment?

Nix lets you build docker images. Those images won’t have Nix program installed, but they will have the exact same binaries as specified in the shell.nix (for reproducibility). You could run production with Nix instead of Docker but it would be less convenient and require additional setup on the servers.

PS. When saying Docker, I mean any OCI-compatible software. Containers are an open standard .

starsep:

Nix is getting popular lately but it has 20 years.

TIL, Nix is indeed 20 years old, damn.

flohoff · 2024-03-22T08:34:04.835Z

NorthCrab:

I encourage you to give Nix a try. I think the most obvious benefits to any developer,

Sorry my friend - you wont convince me that adding something to a solution makes it better security wise or operational wise. It might make it easier for the lazy developer, operationally its a desaster.

Adding complexity is always something to avoid.

In good and secure design removing stuff is the way to go.

And in this case let me point to RFC1925 - The 12 networking truths.

(12) In protocol design, perfection has been reached not when there
is nothing left to add, but when there is nothing left to take
away.

And this applies to ALL technical issues.

Flo

Peter_Elderson · 2024-03-22T09:22:54.628Z

flohoff:

(12) In protocol design, perfection has been reached not when there
is nothing left to add, but when there is nothing left to take
away.

And this applies to ALL technical issues.

So, let’s take away all OSM-tools to get closer to perfection.

Or, add something like “…without harming the required outcome” to this universal truth.

Matija_Nalis · 2024-03-23T01:49:51.688Z

NorthCrab:

Even in 30 years from now on, the same derivation tree will build and run in exactly the same way - isn’t reproducibility great?

I had really hoped for less dystopian future then using patchelf with /lib/libsystemd.so in 30 years time.

That being said, one of the better selling pitches of OSM-NG as I understood it has been that it would be significantly easier for wannabe-contributors to jump in then current situation of having them learn Rails? But if one however instead has to grok Nix first as a prerequisite…

Hashmush · 2024-03-23T13:59:41.565Z

I don’t think anything really prevents you from running it without Nix. It just works as a really good glue to connect all dependencies.

Anyone can help out and provide alternative development instructions, if they’re keen on it. I think @NorthCrab made a mistake by calling it a hard dependency. It shouldn’t really be.