Cutting deployment times by up to 95%

Since we introduced the deployment feature in Structr 2.1, our productivity increased steadily.
Being able to quickly spin up a completely emtpy Structr instance and deploying an existing application without the user data proved to be a very
convenient and fast way to test/develop new features. Not to speak of the other benefits like version control, making it easier to work with multiple developers and so on…

However, with the growing size of Structr applications and growing user data, we saw a drastic increase in deployment times when it came to
deploying new versions of a Structr application to the live environment. One application took a whopping 60 minutes (!) to import - admittedly
on a slightly weak server, but still completely inacceptable downtime for an application.
So I decided to investigate if deployment times could be improved - with awesome results.

The investigation

Using jstack to capture stack traces of a Structr instance during a deployment import I quickly realized that the slow speeds were caused by
a combination of Structrs core features: Cardinality enforcement and Security.

Cardinality enforcement

In the schema editor the cardinality rules for relationships can be defined.
Basically there are four options: OneToOne, OneToMany, ManyToOne and ManyToMany. Everytime a relationship is created, Structr checks if cardinality rules are violated.

Security

Everytime a node is created, two relationships are created. One relationship identifying the owner and one granting the creator rights to the node. Both relationships point to the user who created the node.
The cardinality is OneToMany for ownership (a node can have exactly one owner and a user can own many nodes) and ManyToMany for security grants (a user can have multiple security grants and a node can have multiple security grants as well).

The problem

As one can probably see, creating lots of nodes (like one does when creating pages) leads to a lot of relationships to the user account one uses - probably the ‘admin’ user.
Because security grants are essential for rendering pages differently for different users, these relationships must be included in the deployment export if one wants to transfer
the application to another instance. And therein lies the problem - when importing a deployment export, we have to check if the relationships violate cardinality rules.
If a node has lots and lots of relationships of that type, checking takes a while (and gets slower and slower as more relationships are attached).
For OneToOne, OneToMany and ManyToOne there was a smart strategy in place: Always check from the side of the relationship that represents the “One” part because there can only be one
relationship. For ManyToMany this optimization does not work because both ends can have lots of relationships, so one direction was chosen. This direction happened to be from source to target.
In our specific case this was a problem because during deployment this was the direction User->Node and the user was the node with the ever-growing amount of relationships.

The solution

The solution was as simple as inverting the checking direction to from “source to target” to “target to source”. This works because the check happens at creation time and at that time
the user can already have lots of relationships and the previously created node+relationship can only have a handful of relationships.
With this approach the deployment time (on that specific application) was cut from about 60 minutes to about 3 minutes.
Deploying that application into an empty database took around 300 seconds before and now takes around 100 seconds. A general rule of thumb is: The more data exists in the database, which has grants for a user/group in the deployment export, the more time is saved with the new approach.

One more thing

We further improved things! In more recent versions we added a flag to the User type which can be set for admin users. With this flag set Structr does not create any relationships for ownership and security (for this user) which improves speed and cleans up the exports as well. Simply open the “Edit properties” dialog for the user in the “Security” area and search for “Skip Security Relationships”