Developer Blog

Welcome to a new posting series, developer blogs! These developer blogs will go over information related to the network to keep you in the loop about what's going on behind the scenes, because why not! 

These will contain information on a base level so people can understand what happens, without providing any information truly related to our back-end processes.

> But who knew databases would be the cause of so so so many problems here at Mineshore?

As you may have noticed, over the past couple days, we have had pro-longed periods of downtime, where nothing was working and you couldn't connect or use specific features within the network. However, you never saw what we saw back-end.
When we post a "server maintenance" or "server down for x amount of time to fix things", a lot of the time, it isn't any issue that you can see directly in-game, this is because we have measures in place for if things fail, they fail gracefully. This just means that if something breaks, it doesn't take the network down with it. However, this means that if an issue arises, we won't actually know until we go and check our live console of the network, and only then do we go in to panic mode as we don't know a lot of details: what is causing the error, how long has it been happening, will this impact player data? Those are the kind of things we have been dealing with since release, all to ensure that the server remains stable and live.

But, then what is the actual issue which all of this boils down to? That would be our database(s)! We may not be a large network, but databases are core integral part of our infrastructure here on Mineshore, with many of our custom features, as well as out-sourced features, running on databases (specifically a mix of MySQL and MongoDB databases, as well as a Redis channel) non-stop. In total, before the changes made today, we had a total of 5 different databases spread across 3 different hosting platforms. These databases covered a lot of features: Player claims, Ranks & Permissions, Player warps, and Core database infrastructure. We initially thought this was a good method for our initial release as we are a small network, with not a lot of data being processed. However, this was not the case. 

So, lets get into some statistics so you can see how much data we are actually saving (in terms of player data). In one database, we had roughly over 30 MILLION rows of data points. This may seem like a lot, but it only totaled around ~7 GB of data. However, we had never planned for our databases to reach this size this quickly. Other databases weren't as close, but a rough variation of between 5,000-10,000 rows were present in the rest. As a result of our planning, and the small databases which we were using, once we reached these numbers, our databases failed and caused a massive explosion of different issues.

This includes connection sizes. If you didn't know, you have to preset the amount of allowed connections to a database at one time. Whilst you only see a couple servers when logging into the network, we have many more, but because of this, we were also capping on the amount of times we could connect a server to a database, massively slowing down performance of specific commands and processes (such as store purchases!). Another issue caused by our old database setup was that we had split access. We could only access a small aspect of our databases at a time, and if one of us weren't online and an issue occurred, it would take a while to resolve. 

> So, resolutions anyone?

To resolve a lot of this, we centralized a lot of different processes back end. This is more of a management aspect, which will never impact any of you, it just now means if an issue occurs, everyone that needs to have access to something to investigate and repair an issue, has that access. This also resulted in us now self-hosting and managing our databases. In retrospect, we should have done this from the beginning, but as we move towards getting the server ready for a long-lasting future, and moving away from just getting features up for release, and then repairing any issues from these features in our rush to get the network available to anyone.

We have also changed how much storage is allocated to our databases, therefore increasing the storage available for us to use without the databases failing silently, or the network crashing as a whole. With our new system, we also hope to implement many new features which would've been previously impacted by our infrastructure. In total, we now have near 100 GB of database storage we can use - which seems like a lot, but it really isn't. The reason we aren't concerned about this, is that a lot of the data that we store on these databases will also be wiped on a new season, so we don't have to deal with massive numbers other servers have trouble with. This will help random down-time caused by us not having enough storage, and we have more control over how our databases are actually utilized by the network as a whole. 

We just want to thank everyone who has stuck with us during these turbulent times. From here on out, the network will be more stable, and we will have greater control over what and how we implement new features, which we have a lot planned with. You support does not go unnoticed, and we hope to continue to provide insane experiences, so get excited as we prepare more, and don’t forget to continue spreading the word of Mineshore!