August 25, 2026 · 14 min read · Michael Ludwig

Trust no one, not even 9systems

Storage is one of the most crucial pieces of infrastructure — do you really want to rely on trust when it comes to handling your data?

The first blog article covered the origin and overview of the 9systems idea; this next one tackles an important follow-up question: Why should you trust 9systems? It describes the design the 9systems implementation will follow.

We all rely on many different kinds of infrastructure. It starts with roads, electricity, water, and extends to the internet and the cloud. We trust it, and sometimes we even forget how much trust we put in it. Sometimes, systems suddenly come into focus because they fail where no one had them on their radar (e.g., the Texas blackout, the global CrowdStrike outage, or the OVH data center fire). Some organizations, however, cannot just rely on trust. Navies still teach celestial navigation, hospitals have emergency generators and so do data centers. Now, what does this have to do with storing data?

Running the trust chain

Let's run the trust chain with the goal of spending as little trust as possible. Spending trust is just another way of saying: taking risks. You might start storing your data locally on your premises, which is a good start — but how safe can you really keep it? There are always the factors of malice and competence: You need to have the competence in-house to not make errors, and if you only store in one location, you are exposed to a whole class of location-based risks, like fire, natural disasters, physical attacks, etc.

To mitigate these risks, and to keep your data properly durable, you put it into the cloud. AWS S3 promises 11 9s, which sounds like a great deal. And, as a side note, while this is genuinely good, it's worth being aware of the math here: this is durability per object, so if you have a huge number of objects, it becomes surprisingly likely that the promise allows for losing one over the course of a year (50% chance to lose one object per ~70 billion objects). And it gets worse: these 11 9s are not actually a promise (SLA), but rather AWS' advertised design target. So you really have to pay with trust here. You don't know how the number comes about, or whether it's true at all. There are reasons why in data storage virtually no vendor gives out strong durability SLAs, mostly because the liability would be unbounded compared to the cents-per-GB prices.

Your data being in the cloud has another implication: it is not your computer anymore. So, do you trust the vendor enough to let them see your data? While many individuals don't care, for most organizations, the answer is no — at least for their most valuable data. S3 supports encryption, but you still have to trust that S3 actually encrypts the right way on their side and that they handle the keys responsibly. You could go with end-to-end encryption, but this can be quite the burden on your side and you also have to get it right.

Aside from these very direct trust matters, there are also aspects like jurisdiction: do you trust that the country the vendor is located in doesn't go haywire (think US CLOUD Act)? You can start to use data centers in other parts of the world, but that only helps in some cases, because jurisdiction sticks to the operator, not to the building (which is also why the new "sovereign" regions of the big US clouds don't change much): the CLOUD Act, for instance, reaches data held by US providers no matter where the data center is located. Ultimately, you have to trust in the system consisting of the vendor, the vendor's jurisdiction, the data center, and the data center's jurisdiction. You have to trust that they are competent and that they are not malicious.

Stop spending trust

If you don't want to spend that kind of trust, and do what the navy or your hospital does, then 9systems, an S3-compatible storage system, can help: data is split and distributed redundantly across many S3 backends from different vendors. No backend sees all data, nor can a single backend become a single point of failure. Each vendor only sees pieces of ciphertext, made so that a single piece is useless on its own. So even if they wanted to be malicious, they couldn't do anything with it: they can't read anything, and manipulation attempts would be caught and corrected using the redundant storage. And on the competence side, even if they lose data or fail at integrity, 9systems would catch it, again thanks to redundancy. And, remember the 11 9s you had to take on faith? 9systems won't hand you a durability SLA either, but you can check the durability math you get from the implementation and configuration in detail (see appendix for some napkin math on durability: assume every vendor is 1000x worse than advertised, then the system as a whole can still beat S3's advertised durability). Furthermore, the data dispersion across independent vendors reduces failure correlation. Note that the distribution might look like more moving parts and more complexity — but 9systems spends those on resilience, not fragility: any single backend can fail or vanish without consequence. To be fair, 9systems is not the only product that splits data across clouds. But most others then ask you to trust their platform or their appliance instead, and the rest of this article is about not doing that.

Fig. 1Trust boundaries: ordinary cloud storage, and 9systems.

Don't you now trust 9systems?

So far so good: 9systems saves the day and most of the worries you could have about trusting your data to AWS go away. However: don't you now have to trust 9systems instead? Have you really gained anything? At this point in the story we have already gone through a few steps of the trust chain, and every time some new trust issue emerged that had to be addressed. Let me show you how 9systems lets you win the game of whack-a-mole.

9systems' data handling is about splitting and distributing the data, which dismantles the cloud vendor trust requirements. And 9systems takes itself out of the trust equation by not being a black-boxed service. All infrastructure is deployed in the customer's name (they pay the infra cost directly, so 9systems cannot mark up the cloud bill; it is paid only for operating the service). The only role of 9systems is being the operator and maintainer of the system. You get full transparency from code to deployed resources. Also, as an operator, we never see plaintext: data (plain or encrypted) is never persisted locally; it only lives in RAM. Since that could be accessed, we lock ourselves out: no one, not even us, can SSH into the machines that run the 9systems server. The secret sauce here is immutable infrastructure where change = redeploy. Yet, if change = redeploy, doesn't whoever redeploys hold a lot of power? For that reason, the deploy path is not a black box either: every change reaches your system as a visible commit, built and deployed by a CI/CD pipeline you can inspect. If someone forced us to ship something shady, we would have to show it to you first. That way you don't have to (and you shouldn't!) trust 9systems in quite a few different ways:

Is the data handled competently? You can check for yourself: source code, logs, audit trails, etc.
9systems acting maliciously? We would have to do it in the open: the code is transparent, the infra is yours, and we locked ourselves out of the machines. Nothing happens behind your back.
Jurisdiction? The backends can be distributed all over the world and the 9systems server infra itself is stateless and therefore, in a sense, mobile.
9systems vanishing? Admittedly an issue the big vendors don't face (although Google likes to discontinue services on a whim). It's your infra: the service doesn't stop suddenly, it's just not maintained anymore.
Just unhappy with 9systems? Fire us without losing the data.

A last mole to whack is the need for end-to-end encryption, which you'd want if you don't trust the storage. But it is then your burden, since one of the ends is the client. Because 9systems runs in your infrastructure, it is one of the ends: the encryption end moves into your trust domain, and 9systems handles it for you — competently and transparently. Also, note the order: the data is first encrypted and then encoded into pieces that reveal nothing on their own, which makes it practically impossible to decipher it on any given backend.

What remains

The last possible trust element in the chain is still partly 9systems. It is deployed onto your infrastructure, cloud or on-prem. Even though 9systems operators lock themselves out of the systems that see plaintext and data will never touch local disks, the base cloud vendor's hypervisor could let them read data. But if you're concerned with that level of thoroughness, then standard cloud is not usable for you anyway. That said, there are currently services emerging which put RAM encryption into place, also closing this trust vector. Deploying on-premises instead gives you a managed, auditable, and private service, while itself staying stateless, meaning you can leverage the high durability properties of the cloud.

What also remains is the obvious fact that 9systems also lives within some jurisdiction. Judicial interference can however be seen as a malicious attempt, which we already addressed: 9systems could not act secretly if coerced and won't ever have direct data access.

No free lunch

The bottom line is that if you follow through, 9systems is the systematic way to find a true minimum of trust spending for your data storage. On the flip side, there is no free lunch, so for the sake of a complete picture, what you're trading in is cost and latency. It's up to your judgment.

If you want to dig deeper — or poke holes into any of this — talk to us.

Appendix - Durability napkin math

In 9systems you use a set of cloud vendors to make up this distributed storage. Let's run the math of how the number of 9s of the backends carries over to the offered storage.

Let $n$ be the number of storage backends we use (each piece of data is distributed over these) and $k \leq n$ be how many pieces we need to recover the data. E.g., if $n = 3$ and $k = 2$ , then we can tolerate one corrupt or missing piece to recover the data, or, put differently, we can tolerate one failing backend.

You might already sense the binomial coefficient coming into play when computing the durability, but before doing the calculation, it is important to be precise about what we're actually computing here. When we do this, we assume that failures are uncorrelated. This is an assumption that the cloud vendors also have internally (availability zones, etc.), however we go a step further, because having different cloud vendors is clearly a win for decorrelation.

Another aspect we have to be precise about is that the math is purely about the backend cloud's combined durability. In reality, there is of course also the 9systems layer itself. In the unlikely case that 9systems writes data in an invalid way, it would lower durability, even though all backends might do their job. 9systems also keeps metadata objects and you need them to be retrievable as well. They are stored even more redundantly than the data. At worst this doubles the probability of data loss, which costs a fraction of a single 9, so we will ignore it here.

Having that out of the way, let's focus on one object. AWS S3 "promises" 11 9s that you can still retrieve it after a year; let $d_{backends}$ be the durability of the backend storage (e.g., 0.99999999999); we only use one number here, so you can assume this to be the weakest durability number among the used backends.

d_{9systems} = \sum_{i = k}^{n} (\binom{n}{i}) d_{backends}^{i} (1 - d_{backends})^{n - i}

Here we are summing over all combinations that still let you retrieve the data and for each one you multiply the probabilities for that case. We can simplify this a bit by only considering the worst offenders in the loss probability $1 - d_{9systems}$ (the minimal subset size that can make retrieval fail) and leaving out the surviving backends' factor in these cases, since it contributes very little. This gives us:

d_{9systems} \approx 1 - (\binom{n}{n - k + 1}) (1 - d_{backends})^{n - k + 1}

then, with $(\binom{n}{n - k + 1}) \leq n^{n - k + 1}$ , the coefficient disappears into a pessimistic bound

d_{9systems} ≳ 1 - (n \cdot (1 - d_{backends}))^{n - k + 1}

and finally, since a durability of $d$ means $- \log_{10} (1 - d)$ nines,

# 9 s_{9systems} ≳ (n - k + 1) \cdot (# 9 s_{backends} - \log_{10} n)

And now we can simplify it even further by assuming a number of backends of 10 and below:

# 9 s_{9systems} ≳ (n - k + 1) \cdot (# 9 s_{backends} - 1)

So, the result is, as a rule of thumb, 9systems' 9s grow proportionally to the number of redundant backends. Add a backend for redundancy (increase $n$ , but not $k$ ), you gain its 9s for the 9systems storage. (And yes, that's where the name comes from.)

See the following table for some examples, with the exact durability (from the binomial sum) next to the rule of thumb:

$n$	$k$	9s backends	9s with 9systems (exact)	9s with 9systems (rule of thumb)	Note
1	1	11	11	10	The trivial case
3	3	11	10.5	10	Distribution without redundancy has a mild durability cost
3	2	11	21.5	20	One redundant backend and you double the 9s
5	3	11	32	30	Two redundant backends
6	3	11	42.8	40	Three redundant backends
1	1	4	4	3	One unreliable backend - unusable
5	3	4	11	9	Using a set of unreliable backends gives a reliable storage

The last row is maybe the most interesting one: Even if you don't trust AWS's 11 9s durability claim and you assume it's way lower, combining unreliable de-correlated (different vendors) backends yields good durability again. Also note that 9systems' repair mechanisms, which we ignored in the math, will kick in if one piece of the data got corrupted or deleted. So to actually lose data, sufficiently many backends would have to lose the correct data in a shorter timespan.

At the same time, you can see very high values appear, like in the $n = 6$ case. The math gives us almost 43 nines - and this is where it gets to a point where one should not take these numbers too literally, especially because the assumption of uncorrelated failures certainly breaks down here: at these scales, even very unlikely shared failure modes dominate the arithmetic. So take these numbers with a grain of salt and let them serve your intuition of how to think about the tradeoffs. Ultimately, don't take any vendor's numbers at face value, but reason for yourself.

The bottom line here is that, if you use storage redundancy in 9systems, you don't have to worry as much about the durability of the backends.