In this post I’m covering how I used searchkick and the apartment gems to get a powerful search feature for all the tenants in my Rails 4 app.
For those who are familiar with both gems, you can skip to the “implementation” part.
This gem adds some super powers to Elasticsearch in your Rails application, adding some stemming, special characters and misspelling features out of the box. It has some interesting analyzers and indexes that should be carefully analyzed if you want to fully understand what this gem is capable of.
This gem easily adds multi-tenancy to your Rails app, adapting its strategy to your needs. You can any RDB you want, but this gem really shines with PostgreSQL, where it uses the full capabilities of its schemas feature.
In this gem readme, you can see which is the best strategy for changing database. These strategies are called “elevators”, and they help you to switch databases whenever you want inside your app. I chose the subdomain strategy for the app I was working on, where it made the most sense making many subdomains for each unique username on the app.
The source code of this gem is really great, so writing a custom “elevator” (written as a Rack Middleware) is not hard at all.
After installing Searchkick in your Gemfile, and following the instructions written in the project README, you should notice that your model now have some new class methods, including the most important one: the searchkick_index method.
The searchkick_index method is used all along searchkick, so it can know which Elasticsearch index it should be querying. The searchkick_index method source code show us that we can pass a Proc to the :index_name option, and it will call the block that we passed, allowing us to customize our index name.
So, with those knowledge at hands, we could do something like this:
And that’s it! Every time you change tenants, you’ll have Elasticsearch using a different index with a proper name.
You can create a concern for keeping thins DRY, like this:
Your classes should look like this:
searchkick:reindex:all task?In a project, just giving different index names for your models will not be enough. Like the searchkick README tell us, you should reindex all models in some situations, and now we have indexes for each model/tenant pair! How we’ll do that?
Simple: based on the searchkick:reindex:all source, You can write a task like this one:
Since I’ve got this problem, I was trying to find an elegant way to work with searchkick and apartment. I’ve commented on this searchkick issue, but at the time I ended monkey patching the whole searchkick_index method. Monkey patching this method works fine, but it will completely overwrite this method! We don’t want to mess up with that!
I’ve created a sample project that have a SchemaSearchable concern and a rake task that helps to reindex all tenants using searchkick. This project can be found here
Comments
Marklar ·
Thanks for the helpful post. I'm new to elasticsearch and am implementing now with searchkick in my multitenant app. I've realised that creating an index for every model and tenant pairing is going to be expensive and possibly not scalable (https://www.elastic.co/guide/en/elasticsearch/guide/current/kagillion-shards.html).
Did you implement this in a deployed production environment and if yes, are you able to share how you were able to keep costs down and allow scalability? Thanks.
tiagoamaro ·
Hi Marklar ,
I've used this approach on a real production application that didn't reach more than 30 users, so I didn't have issues around scalability. It kept very reliable and responsive with a two node cluster hosted on http://found.no/. I no longer work on this project, so I don't know how it's going nowadays.
I chose using the User model to separate tenants, since I knew we were going to have "few admin users which would create different stores", so their numbers wouldn't grow so much. It's possible that you have a bigger problem on your hands. Probably, you could solve it by thinking on an additional layer to separate your tenants, and not necessarily separating them by each user, but groups of user (e.g. stores, buildings, neighborhoods).
Marklar ·
Thanks for taking the time to reply. At the moment I'm creating an index per model and using searchkick's where clause to add the tenant filter.
With 30 users you still would have had a large number of indicies created though wouldn't you? For example, if you had 2 models then that would be 60 indicies (at least 60 shards if I'm not mistake). Plus you likely would have replica shards as well.
I'm reading how too many indicies are a bad thing but I'm unsure what constitutes too many. If we look at the plans on Bonsai we see the $50 plan has a limit of 20 shards, 30 users would already require more than this.
For reference, here is some of what I have read:
>It's actually quite common (unfortunately) for many Elasticsearch clients to adopt an index per model paradigm as the default for data organization. Sharding this way can be a premature optimization unless your scaling trajectory is well understood. In practice, small sets of data probably won't benefit from any kind of data partitioning. It’s perfectly reasonable to treat an index as analogous to, say, a Postgres database.
Elasticsearch even supports this kind of internal partitioning with their index “types” feature.
https://docs.bonsai.io/docs/reducing-shard-usage
>ES makes it very easy to create a lot of indices and lots and lots of shards, but it’s important to understand that each index and shard comes at a cost. If you have too many indices or shards, the management load alone can degrade your ES cluster performance, potentially to the point of making it unusable.
https://www.loggly.com/blog/nine-tips-configuring-elasticsearch-for-high-performance/
>There is a limit to how many indexes Elasticsearch can cope with. Every shard of every index is a separate Lucene index, which in turn comprises several files. A process cannot have an unlimited number of open files. Also, every shard has its associated control structures in memory. So, while we will let you make as many indexes as you want, there are limiting factors.
https://www.elastic.co/cloud/as-a-service/pricing
tiagoamaro ·
Thank you for your research around the issue Marklar , but I wouldn't know to answer you precisely, since that project first used a digital ocean droplet, and after that, I migrated to ElasticCloud (which was called found.no). On ElasticCloud, I used 2 two data centers and never looked back to carrying about Elasticsearch setup.
On both scenarios, I never had much complications on supporting all indexes with its documents. The scenario was small: 30 users with about 250 searchable products each one, which leads to 7500 documents of 3KB, totaling 22.5 MB documents within 30 indexes (each index for user/product relationship).
IMHO, I don't think you should worry about the max amount of files and indexes that your application is going to handle. I can't imagine a scenario where, using searchkick + multi tenant approach you would have more than 50k indexes from day 1. Probably, when you reach this point (or if you already using sidekiq as a different approach to an existing app), I would stick with my past recommendation: a higher level of abstraction on your models.
Cheers,
Cosmic ·
I've had a lot of trouble running out of shards. One index per model, per workspace is absolutely a problem. One that I'm not sure how to solve yet but the ES shard limit is my number one issue. This is a highly, highly unscalable solution.
Jordano Moscoso ·
Great post bro. Question, should i add that to the excluded models (public tenant on apartment) too?
tiagoamaro ·
No.
But, if you want your excluded models to be searcheable, you'll have to use a different strategy for naming their indexes (since they're going to be shared between tenants). Probably something that won't include the
Apartment::Tenant.currentstring. In that way, all users could search the model.Jordano Moscoso ·
You are right. I just use
-> { [Apartment::Tenant.default_tenant, model_name.plural, Rails.env].join('_') }And everything works.Gordon DuQuesnay ·
In our app we are doing multi tenancy via default scopes rather than using the apartment gem. So far we've been adding the account_id (tenant identifier) to the index. By doing:
And when querying it including
where: {account_id: Account.current_id}in all queries. I think a similar approach would also work for the apartment gem without having to worry about number of indexes, shards, Bonsai plans on heroku, etc. It seems to be working so far but please let me know if any of you see a problem with this approach.Edit:
Forgot to mention that we also included a method to do unscoped reindexing which we found to be much faster than switching to each account:
joshuapinter ·
Just letting you know, we're doing the same thing. We overrode the
searchmethod on all "searchable" models and just addwhere{ account_id: @account.id }to all search queries automatically. Works great! Thanks for the pointer in the right direction.