Recently someone asked me how I organize my Rails apps. That question came unexpectedly, and I probably did a poor job explaining what I do – this is essentially something that can span multiple hours of discussion, and probably will bring lots of disagreements. So I decided to make a post, and hopefully this might help other people (and to be fair, this is not only valid for Ruby but for any language, really).
First things first, if you I am working in a project with multiple people, I will try to keep the project’s defaults; if the project already have specific organizations (like “app/services” or “app/presenters”) I will follow them; Rails is probably one of the best experiences I have ever seen in collaborating with other teams, and eroding this might bring down the biggest benefit of the framework. But considering I don’t have an already existing structure, or if I need to create some new ones, I will try to bring some personal rules:
Rule 1 – simplify
Simplicity is not a straightforward thing – we might thing something is simple, but it actually is just “small” or “naïve” or things like that. When I say “simple”, I mean that we don’t need to create 5/8 objects to create a single record in the database, or that we don’t need to define a “builder object” when a simple .create! works. So for example, supposing I am writing a system right now that calculates a score for each product I have. If that’s everything that the code does, I would just keep the Rails defaults – app/models for every database, controllers and views, and finally if I need to create a new “score” I would just call the model with .new then .save – and that is it.
But this is where things might become confusing – that’s not all that I do. For example, suppose I implemented an API that allows some pre-filled and already-validated code to inject these on our system (maybe the scoring was done manually by clients, or things like that). In this case, because I control both the system that will call the API and the API itself, I would just use .create! – and yes, that will cause an exception. But in simplicity terms – it’s easier to see our monitoring system show an validation error exception, than check what’s wrong on logs of the system when we find some surveys missing. For some people, that might not be a good practice (by the way, it should never be if your API or interface is client-facing) but it makes things simpler both in terms of code, and also in terms of “finding what went wrong”.
It might be tempting to “future-proof” our code, but in my experience this usually leads to the old “big design up front” – we are almost sure that a component will need to be flexible because of future enhancements we obviously will need, but then when the time comes to the enhancements, they are either very different from what we imagined or, worse yet, the enhancement needs a full rewrite of that part of the system – so all the months (or even years!) that we suffered with a complex architecture to avoid working more in the future were in vain.
The hard part is that simplicity is not always obvious. When I started with Rails, for example, there was no support for foreign keys – and I still think this was a big error. In my own experience, almost all companies I worked with had some data consistency problems (things that could be mitigated by foreign keys, constraints, unique custom indexes, etc). And yes, I did have issues that required me to remove indexes because they were slowing down inserts – two times, to be exact – so my “default” is to “create index” and if that gives me trouble, delete it later. It’s actually faster to do this way too.
Rule 2 – avoid repetition
This is the old “don’t repeat yourself” rule, but I want to break it down a little bit. Supposing this system above grew, and now I need to create users with specific permissions; one kind of user can invite other users, and these need to follow up some specific workflow. Finally, users can be created via a “single sign-on” option like Google or Microsoft accounts, and maybe there’s an automatic flow to bulk create users when a new product is created in our system (and we already have the reviewers for the product). In this case, now the “account creation” became a bit more complicated – but we can still keep it simple!
One solution to that is to create some “workflows” that have some pre-defined things – for example, we can create app/services or app/processes to define an UserCreationProcess or UserCreationService. These will be responsible for opening up a transaction, creating the right elements in the right order in our database, associate everything, validate everything if we are touching multiple models, and bring back some object saying that something is “valid” or “invalid” – maybe in the “invalid” case, we can even mimic how ActiveRecord shows errors such that the UI won’t break (and/or we won’t need to change too much how Rails’ interfaces work). By doing that, we can use a simple UserCreationProcess.call(user: 'data', here: 'with', any: 'details') and that will do the right thing – create an user.
In this case, we keep the simplicity while avoiding code duplication. We can test this class in isolation, and be safe that in each user creation workflow, we won’t miss some detail – even if the details change in the future (maybe we need another attribute to be filled? No worries, every user is created in the same place).
Rule 3 – avoid code that induces errors
In this case above, we made the user process open up a transaction. Now, let’s suppose we have a different workflow that creates a new user, but also needs to create a couple of products and associate the user with said products. There are multiple ways of doing that, but let’s imagine that product creation is also its own “process” because it needs to create some other stuff too and validate some complex conditions.
In this specific situation, I will probably change how these “processes” work – because they open nested transactions, they might partially create something, and because of the way Rails works with transactions, this might even cause some weird and hard to find bugs unless you remember to add some flags to .transaction. So a better way could be a “chaining” rule – for example, instead of calling UserCreationProcess.call, we could define some rules for every process – let’s say, they can return [:ok, <record>] when things worked, [:partial, <record>] when they were partially created, and [:error, <errors>] when nothing was created – and then make a “chain” for these processes. One possible implementation could be:
ChainProcesses
.call { UserCreationProcess.call(.....) }
.and_then { |user| ProductCreationProcess.call(user: ..., other: 'params) }
The and_then implementation will check if the other result had an :ok on it, and if it was, then it’ll propagate the actual record/object that was returned after the :ok to the next “process”. In this case, instead of UserCreationProcess or ProductCreationProcess opening up their transactions, the ChainProcess could open one, safely, rollback when it detects an error, and propagates the error without calling the next and_then chains.
Obviously, there are multiple ways of doing this – this is only one example, and one that I would only do for more complex interactions, but essentially it keeps the “simple” approach (chaining calls is simple, and we don’t need an if inside a user creation to decide if we’re going to create products for that user too, so again it’s easier to test), its somewhat future-proof (we can call ProductCreationProcess for any user, not only new ones), and it’s easy to replace if we need (future-proofing a code might mean “if we need to replace this, make it easy to do so”).
Final thoughts
There’s no “one-size fits all” in software developing. There are a couple of “good practices” but “good” doesn’t mean “every codebase should apply these”.
For example, in one of the cases I had problems with slow indexes was a company that I had to inject a lot of data into an external API. Unfortunately the API was very slow and we had to decide a different way to propagate the data – we first thought about using “batch files” but even that was too slow for our purposes (we tried to deduplicate, aggregate, and essentially avoid sending the external API information that they didn’t need, but even with all the enhancements we still had hundreds of messages each second).
The final solution was to use a table. The table would receive a “updated_at” field, that our API client would read, then a “synced_at” field that the API would fill on their side, and a “payload” which was essentially the JSON that the API accepted. We decided to keep the payload as a JSONB object, and we indexed it with a custom rule that avoided duplications for people with the same document (if the payload referred to a person) or to the same company. JSONB was simply too slow for our purposes – the API was not very well constructed, and the payload was very big (very, very big – I’m speaking of several megabytes big).
We dropped the JSONB, and made a custom index matching the document via regexp. That… is probably one of the worst decisions you can do in a database, but unfortunately, it was the only one that would work in our situation – we could receive multiple messages and use PostgreSQL’s powerful INSERT... ON CONFLICT to insert the fields; that was fast, fulfilled our needs easily, and we could split the “make that insane big payload” in a service and concentrate the updates in a single other service so that it would not hammer the database too much.
Is that good practice? No, it isn’t. But it was what worked, after multiple iterations. The second-best thing was to create a field in the database containing the document, and index that field – but that was adding minutes to our sync process, and in peak hours of the day, our Kafka topics were lagging behind and growing too much.
Hopefully this explains how I usually work – try to do the simplest thing first, if the code starts to duplicate too much or need too many steps to do things, aggregate it in a builder-like pattern, and finally, if even that wasn’t sufficient, refactor in such a way that doing the “right thing” that avoids bugs in your system is the obvious way of writing the code. And finally, be ready to try unconventional things – in some cases, that is what will save you.