Thinking about a personal database (like SOLID) to house all my stuff

I don't have the headspace to get into actually coding anything right now. I'm trying to stay focused on a few other projects for as long as I can mange.

It also feels like one of those things that so big I'm a little freaked out about event thinking about where to start (and how much work it would be). But, that's never really stopped me before once I've actually gotten started with a thing.

To answer your questions:

Does the database have to be persistent?

Yes. Mostly. Kinda. (If I'm thinking of persistent the correct way) My first thought for this is my notes. I've got like 10K of them and use them all the time. I'd want them to always be hot.

Things like images though probably wouldn't need to be fully online. I'm thinking something like a combination of always available thumbnails and meta-data with a link back to something like Amazon's Deep Glacier (https://aws.amazon.com/s3/storage-classes/glacier/) where the full assets are stored, but you don't have instant access to them. It would take some about of time to pull them if you needed the full file.

As to it being persistent on any given device, that feels like an implementation detail. Not that that's trivial, it's just that I've been thinking of this from the end-user side, not the builder side. I think the rule of thumb would be "as long as I can get to the stuff I want when I want, I'm good" (though, I do have an inkling of how hard that actually is to pull off, but that's the dream)

A main key to make explicit is that for most of the types of data I'm thinking of they should be permanent (even if not instantly accessible). e.g. If I'm putting my photos in here, they shouldn't disappear unless I delete them

As to storing data on the network specifically, I can totally see that. Kinda a bit torrent style thing but with your own encrypted data blocks (I think someone built something like that before, but I can't remember)

I'm glad you realized its a bad idea to run code on it, in fact I wouldn't allow anything but HTML and CSS

Yeah, I think the separation of concerns is critical. Though, I'm thinking JSONs too. And, really any format folks want to use. The key being it's just text and not executable code. I realize that can open cans of worms, but I'm thinking about the formatting of stuff and how it would work.

For example, if you have a video and you want to create overlays for it with a given style the idea would be to provide a JSON and then whatever the outside rendering engine is would be responsible for doing the work. (and folks could use different rending engines, so, for example, if there's text based stuff, folks could choose to have that overlay the video or kick it to a screen reader so they can hear it)

There would be some non-text data too. e.g. images and videos. But they also wouldn't be executable.

Yeah, I guess at the high level maybe it comes down to literally, it's just storage and there's another layer that is responsible for doing stuff with whatever is being stored.

Does it have to be persistent? Because that makes the problem much more difficult, because then you have to store it in a way that either shards so things are duplicated but also hope those duplicates stay online which is not at all guarnateed. And the database would keep growing and compression even zstd would only do so much.

Storage is a kicker, no doubt. Carrying on from the earlier note, one thing that I've considered is what if Apple could be convinced to come on board. I have this vision of every iDevice being a coordinated node.

This is, of course, pie in the sky, but I'm in "what if" mode

I'm also thinking of this like some of their plans. e.g. you get 5gb free and then it costs a little for more storage.

That's not great, but it might be a start.

In fact, something like making the storage work on whatever your existing phone/cloud/whatever providers free tier is worth strong consideration. Again, not great since it's a limiter to access, but I'm also thinking phase one as well as the end goal

Two other options come to mind: first, convince some rich ass person to setup a foundation for it. No idea how far that would get because my concepts of money break when I start thinking about really rich folks. Math would need to be done

But, more interesting to me is the idea of a government funding. Like, what if we treated this like roads? That would make it super critical that everything be encrypted to hell and back, but that would be the case anyway.

As before, individual governments would have to be convinced but thinking of this on a 100 year timeline could this turn out to be like a drivers license or other gov. issued ID where one of the items that stored in your database is your official gov id?

User being able to delete what they add would help, but thinking of it more as rental space on a p2p database might end up being an easier problem.

I've been responding as I'm reading instead of reading the entire thing first. So, just got to the p2p DB part and yeah, that's inline with my thinking for at least part of it lol.

Also, I'd totally consider deletion critical to the mix.

Part of my vision for this is like a notebook (or, really a grimoire, which is what I call my notes).

Folks would be able to put whatever they want in it and they'd be able to rip out and evaporate anything they no longer wanted. (Caveat, of course being that if they published something already copies would likely exist in other places and people's databases)

---

I appreciate you taking the time and I hope you feel better. And yeah, I'm happy to kick around ideas on this. And, I'm in no rush, this stuff has been bouncing around in my head for a few years now.

I've never looked at the torrent protocol. (That's going a bit deeper than I've ever done before.) If it doesn't support compression that could be done on the client side right? Or is there something else that goes on there?