Nextcloud 11: The Most Scalable Nextcloud Ever

As part of our push to make Nextcloud the best solution for securing enterprise data, we’ve worked on improving the performance of Nextcloud on large installations like the hundreds of thousands at DeiC. With database load the main factor limiting scalability for many of our customers, we’ve worked to decrease the number of database queries for Nextcloud 11 by up to 85%! This enables customers to have more users on a single server, reducing costs. Users will notice this decrease in server load in reduced load times and up twice as fast data transfers with especially small files.

Speed and scalability

From customers, we often hear that at large scale, the database is the most important bottleneck for scaling up Nextcloud servers. It is not easy to scale up databases as they eat lots of ram and CPU and while we’ve worked on using Redis for caching work results extensively it can’t take away all database work. Our focus has been on reducing the number of separate database calls, making sure we grab the data we need in one go rather than going back multiple times. For this release, the most significant change has been to reduce the number of calls related to getting information about shares to a user.

Shares and storage

Looking across a number of our customers we have concluded that the typical Nextcloud user has between 10 and 50 shares. As shares cause a significant number of database queries with every check for changes on the server, this was our focus. The consecutive changes resulted in a significant decrease of the number of queries and the time they take to execute.

Below some statistics of a ‘depth 0 Propfind’, one of the most common operations on a Nextcloud Server. This is the operation clients use to check if there have been any changes and they create up to 99% of the database load on a Nextcloud server!

Less database queries not only results in lower server load but also faster results. We have tested the effect on small (10 byte) file upload and download operations. This represents an optimistic scenario for these improvements as, with large files, Nextcloud simply saturates the bandwidth of a connection. But uploading small files is a significant bottleneck and this has become much faster.

Test setup and generalization

All tests were ran against a server with a local MariaDB database and using Redis as cache. As such a local database has a very low latency and very good performance, it is to be expected that on large installations the advantage of Nextcloud 11 over 10 will be even larger. Of course, as the graph shows, it depends heavily on the number of shares users typically have. If you would run this test on a huge instance where users have dozens to hundreds of shares accumulated over years of collaboration with others and provided the database is indeed the bottleneck in scaling up you can expect this system to easily handle perhaps an order of magnitude more users with Nextcloud 11.

Other improvements

The changes to database scalability are one among a number of enhancements meant to more efficiently use Nextcloud server hardware. Other enhancements include:

• A series of improvements to the handling of previews decreases server load. Generation can be handled by an app at low-traffic times as determined by the system administrator. Storage space efficiency has improved as well by sharing previews between users.
• To make Nextcloud systems with object storage more scalable, this release introduces a multi bucket object store. Not all s3/swift implementations like having multiple million objects in a bucket, so Nextcloud 11 will create a bucket per user, helping with storage performance on large installations. When the storage system used doesn’t like to have too many buckets (Amazon S3), the number can be limited.
• Thanks to improvements in the integration with Collabora Online, startup time has been improved in Nextcloud 11 and users will now be able to enjoy their online office suite faster than ever!
• Large folders will load considerably faster due to optimizations in the Files app