This article isn’t specifically about Solid Cache or encouraging you to use it. Recently, I read the source code of the solid_cache gem and learned some interesting things that I’m going to share here.

Solid Cache is a database-backed cache storage system. In simple words, it enables you to use any database for caching, rather than relying solely on RAM-based stores. Solid Cache was introduced at the RailsWorld 2023 Conference’s Keynote, and Donal McBreen, the main contributor, provided more details about it. I highly recommend watching the presentation if you want to understand the reasoning behind this paradigm shift.

I’d like to thank Donal for reviewing this article and providing feedback, which I have incorporated into the current version.

Before exploring how Solid Cache works under the hood, it’s useful to imagine what a basic cache table might look like. A minimal database-backed cache could simply have two columns — key and value — where each key uniquely identifies a cached object, and the value stores the serialized data. Optionally, a created_at or expires_at column can be added to help the system identify and remove stale entries over time. In such a design, cache lookups are done with a straightforward SELECT value FROM cache WHERE key = ?, and new entries are written with an INSERT or UPSERT. While simple and easy to implement, this naïve approach doesn’t look enough.

Now let’s see what other details have been considered by Solid Cache and what was the rationality behind them. I’ve split the article into sections to make it easier to read and to let you skip each part if you want.

“byte_size” column

The byte_size column is an estimation of the size of each record in the table, including the attributes and any related overhead. You might expect the size to be similar for all records, but in reality, the value can vary greatly from row to row. Solid Cache uses this column to estimate the total size of the cache (I’ll cover that later).

“key_hash” column

The key_hash column stores a 64-bit hash of the key. There are a few reasons for this: since the index is built on this column, and its size is always 8 bytes, the index size stays small regardless of how long the actual key is (even with considering the overhead of having a new column). Integer-based indexes are also faster in most databases. There’s another use case for this column: selecting random rows for size estimation using a sampling algorithm (I’ll cover that later).

Why no Index for “created_at” column?

Since Solid Cache is a First In, First Out (FIFO) cache, it needs an efficient way to remove the oldest rows. The query to find the oldest rows involves filtering by the created_at column. Without an index, this query could be slow. However, adding a new index isn’t free—it comes with storage and write costs. Since we’re using an auto-incrementing primary key, the row with the highest id was created most recently and has the highest created_at value. So, to find the oldest rows, we can just look for the rows with the smallest ids.

Why “key_hash” is part of two indices?

If you look at the indices, you’ll see two that involve the key_hash column:

index ["key_hash", "byte_size"], name: "index_solid_cache_entries_on_key_hash_and_byte_size"
index ["key_hash"], name: "index_solid_cache_entries_on_key_hash", unique: true

You might ask: if we have a multi-column index on key_hash, byte_size, why do we also need a single-column index on key_hash? Or vice versa?

The single-column index on key_hash enforces a uniqueness constraint. Because Solid Cache uses upserts to write entries, most databases require a uniqueness constraint on the target column(s) specified in the ON CONFLICT clause.

The multi-column index helps queries like the following, which estimate the table size, run more efficiently.

SELECT SUM("byte_size")
FROM "solid_cache_entries"
WHERE "key_hash" >= ?AND "key_hash" < ? AND (byte_size <= ?)

When an index covers all the columns a query needs, the database can answer the query entirely from the index, without reading the actual table rows. This is called a covering index (Index-only Scan) and it improves performance by avoiding extra I/O. To learn more, check out the article below:

How We Optimized a Query to Run 1 Billion Times a Day at Doctolib

Calculate(estimate) the cache size

Solid Cache uses sampling to estimate the total cache size. The sample size can be set by size_estimate_samples in the cache.yml file.

Before diving into how estimation is done, let’s look at some alternative solutions that could be considered:

• Using Database Statistics: Some databases like Postgres and MySQL store table and index sizes in their metadata. But these queries are different for each database, and some databases might not support this at all. Since Solid Cache aims to be compatible with any database Rails can use, this isn’t a practical solution.
• SQL SUM() function: You could sum up the byte_size column with SELECT sum(byte_size) FROM solid_cache_entries;, which works in all relational databases. However, this would require a sequential scan of the table, which is expensive if the table is large. Since we only need an estimate, running an expensive query frequently isn’t worth it.
• Estimate size by row count: One simple solution is to estimate the table size based on the number of rows. It seems this was actually the initial approach in Solid Cache. I can guess why it’s dropped, it’s hard to pick an average row size that works for everyone, especially since cache entries can differ widely in size.

Now let’s look at the current implementation. Ignoring some details for clarity, the high-level algorithm is:

def size
  outliers_size + non_outlier_estimated_size
end

outliers_size calculates the size of the N largest rows in the table, where N is the sample size. non_outlier_estimated_size picks N random rows (excluding the outliers) and sums their byte_size values. With this, the algorithm gets a more meaningful average row size and estimates the total table size by multiplying this average by the row count.

To improve accuracy, Solid Cache can keep some of the previous estimates and average them with the new estimate.

Selecting random N rows efficiently is tricky, and I’ll explain that later in this article.

Random Selection for Size Estimation

To select random rows from a table, you might first think of using the database’s built-in random functions. For example, in Postgres:

SELECT * FROM solid_cache_entries ORDER BY RANDOM() LIMIT 10;

Postgres requires a sequential scan of the whole table for this, which is very expensive for large tables. There are some other functions that don’t have this problem, but different databases have different solutions. We want a method that works everywhere.

Earlier, we saw that key_hash is a 64-bit integer, meaning its values range from -2⁶³ to 2⁶³-1. If the hash function is good and the keys are well-distributed, these values should be spread fairly evenly across this range. To select random rows, we can pick a random sub-range within the key_hash interval and select rows whose key_hash falls into that range.

For example, if our sample size is 1000 and there are 2000 rows in the table, to select 50% of the rows randomly, we could select all rows with key_hash between 0 and (2⁶³ - 1). This technique lets us efficiently sample rows for estimation.

How Sharding works in Solid Cache?

Solid Cache also supports sharding, which means using multiple databases to store the cache. For this, we need an algorithm that deterministically assigns a key to a database node. The same key should always map to the same node, even if multiple instances of your app are running at the same time or restarted.

Solid Cache builds an array of length TABLE_SIZE (2053 by default), where each cell points to one of the database nodes. For example, with 3 databases, each cell holds 0, 1, or 2, and each node’s number appears about equally often.

To find out which node to use for a given key, we use:

lookup[quick_hash(key) % TABLE_SIZE]

Here, lookup is the array, and the hash of the key determines which cell to check, which in turn tells us which database node to use.

How lookup table is built?

At first glance, you might try to build a lookup array by simply shuffling the node numbers, like this:

lookup = [0, 0, ..., 1, 1, ..., 2, 2, ...].shuffle

Why cann’t we just use this? Because this isn’t deterministic — every app instance might generate a different lookup table. We could persist the array somewhere, but that’s cumbersome and it only solve this issue. Also, this approach isn’t consistent: if we add or remove a node, nearly every key would be reassigned to a different node.

Solid Cache uses the Maglev algorithm to generate a consistent lookup table. Maglev ensures that if nodes are added or removed, most keys continue to map to the same node as before, minimizing cache misses.

How does Maglev solve the problem?

1. Start with an empty array.
2. Each node gets to pick which slot it wants, in turn. This is repeated until all slots are filled.
3. Each node’s choice is determined by a deterministic formula, so every app instance generates the same table.

The following diagram shows how the algorithm works in a graphical way. The numbers on the arrows indicate the order of actions.

Easy! Now the only thing that we need is a deterministic pick from every node for step 2. Here is my favorite part of the algorithm:

preferred_slots = TABLE_SIZE.times.map { |i| (i * skip) % TABLE_SIZE }

It’s mathematically provable that with this algorithm, if TABLE_SIZE is a prime number and skip is an integer such that 0 < skip <= TABLE_SIZE, then the preferred_slots array will contain each number from 0 to TABLE_SIZE - 1 exactly once, but in a pseudo-random order. This means every slot is used only once, and the order is determined by the algorithm.

Each node uses its own preferred_slots array when it’s their turn to choose a slot. Also, each node needs to track which index in their preferred_slots array they last used, so that they can continue where they left off during the next round of selection.

To ensure that every node’s preferred_slots array is unique, we introduce another variable, offset, into the algorithm. The final code looks like this:

offset = md5(node_name, :offset) % TABLE_SIZE
skip = md5(node_name, :skip) % (TABLE_SIZE - 1) + 1

preferred_slots = TABLE_SIZE.times.map { |i| (offset + i * skip) % TABLE_SIZE }

With this algorithm, we can build a consistent lookup table. The diagram below shows what happens when a new sharding node is added:

After adding node 2, some slots have changed to 2. The diagram also shows that some inconsistencies can occur (for example, a 1 has become 0). This means the algorithm is not perfect, but small changes in other slots are acceptable. After adding the new node, the frequency of node numbers in the lookup array becomes almost balanced again.

How to delete the expired rows?

As mentioned earlier, Solid Cache doesn’t define a database index on the created_at column, relying instead on the id column. Suppose cache entries expire after 2 months — rows older than that should be removed.

Since smaller ids are the oldest, we can look for rows with small ids and check if their created_at is older than 2 months. However, the check for created_at is not part of the SQL command and it’s done in memory after fetching the rows. Why is this?

If we add a WHERE created_at < ? clause to the SQL statement, like so:

SELECT id FROM solid_cache_entries
WHERE created_at < ?
ORDER BY id
LIMIT ?

the database will likely use the index on id to fetch the required number of rows as specified by LIMIT. However, if there aren’t enough qualifying rows to satisfy the LIMIT, the database may need to scan the entire table to perform the created_at check, which can be inefficient.

Conclusion

This article isn’t tied to any particular technology stack, and it doesn’t matter whether or not you use or will use Solid Cache. I found some very cool ideas in the Solid Cache source code and thought they were worth sharing.

I’ve found that reading source code and sharing what I learn is both useful and interesting, and I hope to continue doing it.

What I Learned from Digging into the SolidCache Gem was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

Over the past few months, our team at Doctolib has been working on migrating authorization data to a centralized table within its own database. This authorization database is powered by PostgreSQL. As part of the migration, we needed to transform the schema, which necessitated rewriting the queries. Given that authorization data is frequently queried, it’s crucial that these new queries are both fast and efficient.

In this article, we won’t be discussing the migration process itself; instead, we’ll focus on how we enhanced the performance of queries in the new table.

The following code retrieves the access grants for a single account:

Grant.where(
  role_id: [
    '96761ffd-ccb9-46c8-abb5-5810d0d202bb',
    '06fa4f0b-24de-4f3d-8803-a917606a0fb3',
    '17b47072-8218-4b8c-afc9-f9fcddf1bffe', 
    '62bae27a-4b7f-4999-ac3b-de73e75e8273', 
    '74d99f9a-a04e-408e-a8c7-b1f7c9da4280',
  ],
  account_id: ['123456']
)

The code above generates the following SQL query to retrieve records from the grants table.

SELECT *
FROM "grants" 
WHERE "role_id" IN (
 '96761ffd-ccb9-46c8-abb5-5810d0d202bb',
 '06fa4f0b-24de-4f3d-8803-a917606a0fb3',
 '17b47072-8218-4b8c-afc9-f9fcddf1bffe', 
 '62bae27a-4b7f-4999-ac3b-de73e75e8273', 
 '74d99f9a-a04e-408e-a8c7-b1f7c9da4280'
) AND "account_id" IN ('123456')
-- AND "expires_at" IS NULL OR "expires_at" > NOW();

Note: The actual query also includes a condition for the expires_at column, which we have omitted from our analysis for the sake of simplicity.

EXPLAIN the query

We begin by using the EXPLAIN command. EXPLAIN is a PostgreSQL command that displays the execution plan selected by the query planner for a given SQL statement, without executing it.

EXPLAIN
SELECT * FROM ...

The command above provides the following result:

[
  {
    "Plan": {
      "Node Type": "Index Scan",
      ...
      "Index Name": "index_grants_on_account_id",
      "Relation Name": "grants",
      "Alias": "grants",
      "Index Cond": "((account_id)::text = ANY ('{123456}'::text[]))",
      "Filter": "(role_id = ANY ('{96761ffd-ccb9-46c8-abb5-5810d0d202bb,...}'::uuid[]))"
    }
  }
]

As we can see, PostgreSQL uses the index_grants_on_account_id index to locate records with the specified account ID, ensuring that the database uses an index scan rather than a sequential scan.

Reduce the Data Transferred over the Network

Fetching records from the database involves more than just executing the query itself. Each query execution also includes waiting for a connection, transmitting the SQL command over the network, receiving records over the network, and more.
Therefore, retrieving large volumes of data from the database can be slow. By reducing the response size, we can enhance query performance.

To achieve this, we need to benchmark the query on the application side. The code below measures the execution time of our query.

time = Benchmark.measure do
  Grant.where(
    role_id: [
      '96761ffd-ccb9-46c8-abb5-5810d0d202bb',
      ...
    ],
    account_id: ['123456']
  ).load
end

puts "Time elapsed: #{time.real} seconds"
  # => Time elapsed: ~0.2 seconds

To reduce the size of the retrieved data, we can fetch only the columns that are necessary. To do this, we specify the required columns in the query.

time = Benchmark.measure do
  Grant.where(
    role_id: [
      '96761ffd-ccb9-46c8-abb5-5810d0d202bb',
      ...
    ],
    account_id: ['123456']
  )
  .pluck(:account_id, :resource_id, :resource_type, :role_id)
  .load
end

puts "Time elapsed: #{time.real} seconds"
  # => Time elapsed: ~0.1 seconds

Note: If you’re using an ORM, you can save time by avoiding the initialization of model objects. For instance, in ActiveRecord, using pluck instead of select can prevent unnecessary model object initializations.

More Advanced Aspect of Query Plan

Now, let’s dive into the query plan and explore more advanced features of the EXPLAIN command. We can enhance EXPLAIN to obtain a more detailed query plan.

set track_io_timing = on;
EXPLAIN(ANALYZE, TIMING, BUFFERS)

SELECT "account_id", "resource_id", "resource_type", "role_id"
FROM "grants" 
WHERE "role_id" IN (
  '96761ffd-ccb9-46c8-abb5-5810d0d202bb',
  '06fa4f0b-24de-4f3d-8803-a917606a0fb3',
  '17b47072-8218-4b8c-afc9-f9fcddf1bffe', 
  '62bae27a-4b7f-4999-ac3b-de73e75e8273', 
  '74d99f9a-a04e-408e-a8c7-b1f7c9da4280'
) AND "account_id" IN ('123456')

Let’s take a quick look at the settings we’ve added to the EXPLAIN command:

• ANALYZE Actually executes the query instead of just showing the plan, which shows real execution time and row counts (Total Cost, Actual Total Time, Actual Rows).
• TIMING Shows the time spent in each node of the query plan by measuring CPU time for operations.
• BUFFERS Shows buffer usage statistics by displaying Shared, Local, and Temp buffer Hits and Reads (Shared Hit Blocks, Shared Read Blocks).
• TIMING Seeing execution time for each block could give a good sense to the execution speed for the query [List of the fields that is being added].
• track_io_timing = on Enables tracking of I/O timing statistics by measuring actual time spent on disk I/O operations. It helps us to understand how much time we spent to read non-cached records (I/O Read Time)

[
  {
    "Plan": {
      "Node Type": "Index Scan",
      ...
      "Index Name": "index_grants_on_account_id",
      "Relation Name": "grants",
      "Startup Cost": 0.43,
      "Total Cost": 696.92,
      "Actual Startup Time": 2.824,
      "Actual Total Time": 95.762,
      "Actual Rows": 2882,
      "Actual Loops": 1,
      "Index Cond": "((account_id)::text = '123456'::text)",
      "Rows Removed by Index Recheck": 0,
      "Filter": "(role_id = ANY ('{96761ffd-ccb9-46c8-abb5-5810d0d202bb,...}'::uuid[]))",
      "Rows Removed by Filter": 1,
      "Shared Hit Blocks": 1,
      "Shared Read Blocks": 765,
      "I/O Read Time": 93.775,
      "I/O Write Time": 0.000,
      ...
    },
    "Planning": {
      ...
    },
    "Planning Time": 0.438,
    "Triggers": [
    ],
    "Execution Time": 96.019
  }
]

Note: The attributes not discussed in this article have been removed from the query plan above.

As we can easily recognize, most of the time is spent on I/O Read Time. When the required blocks are not found in the cache, PostgreSQL fetches them from the disk. In other words, Shared Read Blocks and I/O Read Time are directly correlated. Each Shared Read Block typically requires an I/O operation, and more Shared Read Blocks generally result in more I/O Read Time.

How can we reduce Shared Read Blocks?
Shared Hit Blocks refers to the number of blocks read from the cache. The total number of required blocks for our query is 766 (Shared Hit Blocks + Shared Read Blocks). It's important to note that PostgreSQL's "Shared Buffer Cache" operates based on blocks (defaulting to 8KB pages). Therefore, by increasing the cache hit ratio, the total number of read blocks remains the same and if Shared Hit Blocks is increased, Shared Read Blocks will be reduced.

If we execute the EXPLAIN command a second time, Shared Read Blocks and I/O Read Time typically decrease, leading to a significantly reduced Execution Time (which explains why the query runs faster on subsequent executions). When Shared Hit Blocks is much greater than Shared Read Blocks, we describe this situation as having a high cache hit ratio.

If Shared Read Blocks changes with each query execution, we need a method to determine the cache hit ratio for our query in production. To assess the cache hit ratio, we used the Datadog | Database Monitoring tool. In the "Query Metrics" section, we searched for our query and examined the Shared Blocks Hit Ratio column in the table.

After examining the cache hit ratio, we realized it was very low for our new query in the new table. We used two strategies to improve the ratio for our query.

Warming-up the Cache
Warming up the cache involves executing the query more frequently in your database. Since the old query was heavily used in the application, immediately switching to the new query in the new table and database could cause issues due to the cache not being warmed up yet. Fortunately, we were using experimentation, which allowed us to warm up the cache by slowly increasing the experimentation percentage.

Improve Cache Hit Ratio
As mentioned earlier, in our example, the query needs to fetch 766 blocks. We understand that the cache size is limited and cannot hold the entire database in memory. By reducing the number of required blocks, we can improve the cache hit ratio, as fewer blocks need to be cached.

Let’s try retrieving fewer columns from the database by removing resource_type and executing the query again:

set track_io_timing = on;
EXPLAIN(ANALYZE, TIMING, BUFFERS)

SELECT "account_id", "resource_id", "role_id"
FROM "grants" ...

If we execute the EXPLAIN command, we will see that the total number of blocks remains exactly 766, just as before?! 🤔

Why? This makes sense when we consider how records are stored and cached. Each record, along with its attributes (though this can differ for wide records), is stored in blocks or pages. When PostgreSQL needs to read data from a block, it reads the entire block, not just the specific data or columns requested.

What is “Index Only Scan”?
When PostgreSQL can retrieve all required data directly from the index without accessing the actual table, it uses an “Index Only Scan” instead of an Index Scan. This is the most efficient type of scan when applicable. Therefore, we need to define an index that includes all required data. “All required data” means both the selected columns and the columns that are part of the conditions. Before proceeding to define the index, let’s take a look at the complete query, including the expires_at condition that we previously omitted for simplicity.

SELECT "account_id", "resource_id", "resource_type", "role_id"
FROM "grants" 
WHERE "role_id" IN (
  '96761ffd-ccb9-46c8-abb5-5810d0d202bb',
  '06fa4f0b-24de-4f3d-8803-a917606a0fb3',
  '17b47072-8218-4b8c-afc9-f9fcddf1bffe', 
  '62bae27a-4b7f-4999-ac3b-de73e75e8273', 
  '74d99f9a-a04e-408e-a8c7-b1f7c9da4280'
) AND "account_id" IN ('123456')
  AND expires_at IS NULL OR expires_at > NOW()

According to the query, “all required data” refers to five columns: account_id, resource_id, resource_type, role_id, and expires_at. An index with five columns could result in a large index. Let’s test this locally. I inserted approximately 11 million records into my grants table, which has 10 columns. I checked the table size, and it's 1.27 GB. Now, I want to try the following index with these five columns:

CREATE INDEX index_grants_on_five_columns
  ON grants
  USING btree (account_id, expires_at, role_id, resource_id, resource_type);

The new index’s size is approximately 700 MB. While indices provide benefits, they also come with costs — the larger the index, the higher the cost. If this index were the only one for the table, it might be worthwhile to keep it, but that’s usually not the case.
We were able to remove two columns from the query; let’s see how!

Remove resource_type column
In our application, we know that resource_type can be derived from role_id, meaning we don't need to retrieve it from the database; instead, it can be calculated and appended to the result in memory. This is possible because each role_id corresponds to a single resource_type.

Remove expires_at from the condition
The expires_at column specifies the expiration time for a record when set. We also had a nightly background job to clean up records that expired the previous day. Our application logic can tolerate returning a recently expired record, so this condition doesn't need to be very strict. Therefore, we decided to remove expires_at from the query and run the cleanup background job more frequently. Since it's a very lightweight job, it can even run every minute.

After reducing the number of required columns to three, we defined the new B-tree index on account_id, role_id, and resource_id:

CREATE INDEX index_grants_on_account_role_resource_id 
  ON grants 
  USING btree (account_id, role_id, resource_id);

Note: Since the query condition is based on role_id and account_id, these columns should come first in the index; otherwise, PostgreSQL won't be able to use the index effectively. The order of columns is important when creating a B-tree index.

Update: At the time of implementation, our team was not aware of PostgreSQL’s INCLUDE parameter for indexes. In our example, since resource_id is not used in the query condition, it can be included in the index as a non-key column.

CREATE INDEX index_grants_on_account_role_resource_id 
  ON grants 
  USING btree (account_id, role_id)
  INCLUDE (resource_id);

Using INCLUDE can reduce the index size. I tested it locally with approximately 11 million records, and the index size decreased from 700MB to 500MB.

Here is what the new query looks like:

SELECT "account_id", "resource_id", "role_id"
FROM "grants" 
WHERE "role_id" IN (
  '96761ffd-ccb9-46c8-abb5-5810d0d202bb',
  ...
) AND "account_id" IN ('123456')

And let’s run EXPLAIN on our new query:

[
  {
    "Plan": {
      "Node Type": "Index Only Scan",
      ...
      "Index Name": "index_grants_on_account_role_resource_id",
      "Relation Name": "grants",
      "Total Cost": 13.14,
      "Plan Rows": 183,
      "Plan Width": 31,
      "Actual Total Time": 0.875,
      "Actual Rows": 2882,
      "Actual Loops": 1,
      "Index Cond": "(account_id = '123456'::text)",
      "Rows Removed by Index Recheck": 0,
      "Filter": "(role_id = ANY ('{96761ffd-ccb9-46c8-abb5-5810d0d202bb,...}'::uuid[]))",
      "Rows Removed by Filter": 1,
      "Shared Hit Blocks": 183,
      "Shared Read Blocks": 0,
      "I/O Read Time": 0.000,
      ...
    },
    "Planning": {
      ...
    },
    "Planning Time": 0.200,
    "Triggers": [
    ],
    "Execution Time": 1.109
  }
]

With the new index, only 183 blocks were read, and the "Node Type" changed to "Index Only Scan" 🎉. Additionally, the Execution Time has significantly decreased for this specific query. Before celebrating our achievement, let's check the average (or even better, the p99) execution time for this query in production.

After using the new query and index for a while in production, we checked Datadog monitoring.

The Shared Blocks Hit Ratio is 100% 👌 and the Average Duration is 44 μs 🤯. It's time to celebrate!

Conclusion

In this article, we explored how to analyze a query to understand its performance and cost, and we learned several techniques to enhance them.

Is it worth optimizing a query to this extent, or is it overkill?
It depends(as usual)! Are you optimizing the query to improve the response time for the end-user, or do you want to reduce the load on your valuable database? Usually, the end-user cannot perceive a difference of a few milliseconds in response time, so saving milliseconds doesn’t necessarily enhance user experience. However, if the query is frequently executed, saving milliseconds becomes very valuable for the database. In some cases, you might even make your query faster at the database level but slower at the application level because application-level scaling is typically easier to achieve horizontally than database scaling. Scaling a database is more complex, expensive, and comes with certain limitations.

How We Optimized a Query to Run 1 Billion Times a Day at Doctolib was originally published in Doctolib on Medium, where people are continuing the conversation by highlighting and responding to this story.

This article explains how we can create database views in Rails applications. To see a practical usage of database views, please refer to the article below:

Database View in Ruby on Rails

Suppose we have Post model in our application and the table looks like this:

And we want to create a Database view that expose only id and body columns. This can be achieved through 3 simple steps:

1. First, we generate a migration file:

rails g migration SimplifiedPosts

2. Add up and down methods to the migration file:

class SimplifiedPosts < ActiveRecord::Migration[7.0]
  def up
    execute <<~SQL.squish
      CREATE OR REPLACE VIEW simplified_posts AS
        SELECT
            "id",
            "body"
        FROM public.posts;
    SQL
  end

  def down
    execute <<~SQL.squish
      DROP VIEW IF EXISTS simplified_posts;
    SQL
  end
end

In this migration file, up method creates the view. If a view with the same name exists CREATE OR REPLACE VIEW replaces the existing one with the new one (it can also be used to update an existing database view). down method is used for rolling back the migration and in this case it drops the created view.

3. Now by running rails db:migrate the view will be created in the database.

Use ActiveRecord to query the View

Fortunately, we can utilize ActiveRecord to query the view, which makes our life much easier. To achieve this, we just need to create a regular model class:

class SimplifiedPost < ApplicationRecord
end

Now we can query the view by ActiveRecord model class:

SimplifiedPost.find_by(id: 1)
SimplifiedPost.where('id = ?', 1)

Database Views — under particular conditions — are updatable!!! That means DELETE FROM simplified_posts deletes all the records from posts table. For Postgres, look at Updatable Views section for more details.

If you want to prevent manipulation through the model, the simplest approach in Rails is to define (or override) the readonly? method for the model.

class SimplifiedPost < ApplicationRecord
  def readonly?
    true
  end
end

When you mark the model as read-only, it prohibits any actions such as create/update/delete/destroy as well as any methods that modify data:

# ERROR: It prohibits creation of a new record

SimplifiedPost.create body: 'Body for read-only model'
# => SimplifiedPost is marked as readonly (ActiveRecord::ReadOnlyRecord)

WARNING: Since readonly? is an instance method, The class-methods like delete_alland update_all will bypass the read-only check!

Switch schema format to SQL

If you’re using schema.rb format to represent your database schema in your codebase, the created View will not be stored in schema.rb file. Therefore, if you initialize the database using rails db:schema:load, the View will not be created. To have the view in database, we will need to run that specific migration again!

If we change the schema format to SQL, the structure.sql file can accurately represent all aspects of the real database. To switch the format, you only need to follow two simple steps. First, add the following line to the application.rb file:

module RailsApplication
  class Application < Rails::Application
    ...

    config.active_record.schema_format = :sql
  end
end

Afterward you should run rails db:schema:dump to generate structure.sql file, which will be based on the current state of the database. At this point schema.rb file can be removed.

Take a look at the generated file — structure.sql. It should represent the database in SQL format.

Another way: Using scenic

scenic is a gem that lets you to create and mange database views using Ruby methods. It also allows you to keep your schema file in Ruby format (schema.rb). Rather than repeating scenic’s documentation here, I encourage you to check out its Github page.

Conclusion

In this article, we learned how to create a database view in Rails applications, how to query it using ActiveRecord, and how to regenerate the database, including the views.

Is it better to use a gem to manage views? As always, it depends! Adding a gem to your project is not free, it requires ongoing maintenance (like updating it). Or If you’re going to have more complex schema, with things like Database Constraints or database specific configurations, switching to SQL format might be a better option. However, if you’re comfortable with scenic and want to use its additional features, then go for it

Database View in Ruby on Rails

Two months ago after dockerizing our applications, I got a task to implement zero downtime for our deployments. When you don’t use Docker, in order to have 0-downtime, you can use phased-restart feature of Puma HTTP server. With Docker infrastructure, also you can use Swarm or Kubernetes, but most of the times, Swarm or Kubernetes are overkill for us.

In this article, we’re going to implement 0-downtime deployment with some simple tools that you can substitute them with other alternatives, so we don’t want to stick to some specific tools. Keep in mind that, implementing a proper way for container deployment, needs a good knowledge of Docker technology, so before reading this article ensure that you have enough knowledge in Docker.

At first, we will implement a simple Rails application with a single endpoint, and after dockerizing it, we will use Capistrano to have a simple deployment based on the docker containers. Then we will setup Nginx reverse-proxy and in order to do that, we will use nginx-proxy docker image. In the last two steps, we will improve our deployment scripts to support zero-downtime, and finally, we will test zero-downtime on the deployments with ApacheBench tool.

I’ve also prepared a sample project according to this article and you can check it out through the link below:

AliSepehri/zero_downtime

Initialize application and implement a simple endpoint

In order to have a simple application, we want to use Ruby-on-Rails framework. If you’re using other languages/frameworks, the only thing we need to have for this section is a /ping endpoint which returns a response with 200 HTTP status.

In this article, we don’t want to address installing Ruby, Ruby on Rails, and the other prerequisites. I assume that we’ve installed Ruby on Rails on our machines and for initializing a Rails application, we execute the command below:

rails new zero-downtime --api

For implementing /ping endpoint, generate ping controller with Rails generators:

rails g controller ping

Add a simple test to test/controllers/ping_controller_test.rb file, add the route to config/routes.rb file, and the related action to app/controllers/ping_controller.rb :

In the end, run the test to be sure that everything is working correctly:

rails test test/controllers/ping_controller_test.rb

Dockerize the application

Now we’re going to write a simple Dockerfile for our Rails application. If you’re using another framework, for sure you can find many articles on the Internet about the way that you can dockerize it.

Now build your Docker image by running the docker build command:

docker build -t zero-downtime .

Create and run a container based on our Docker image:

docker run --rm --name zero-downtime-app -p 3000:3000 zero-downtime

And test it:

curl http://localhost:3000/ping

# ===> {"message":"pong!"}

If you’re getting the {"message":"pong!"} , it means your container works; stop the running container(Ctrl+C) and follow the article. In the next section, we’re going to deploy the container to our local machine by the Capistrano tool.

Simple deployment with Capistrano

Now we want to use Capistrano to implement a simple deployment flow for our application. For sure there are many alternatives for the Capistrano and you can use them easily because the commands which we have used in Capistrano tasks are very close to the raw shell commands.

At first, we’re going to add Capistrano to our Gemfile, install it and initialize it. In order to that we can execute the following commands:

# Add and install the Gem
bundle add capistrano

# Generate default config files
bundle exec cap install

I’ve shown my Capfile after removing unnecessary lines:

Into the ./config/deploy.rb file, we need to set repo_url to refer to our code. For this, you should use a git server. For example, you can use a Github repository and then copy the ssh remote URL from the repository page to our deploy.rb file.

As you can see we also have some tasks which will be executed after the default deployment steps. We will implement the tasks inside the docker.rake file.

With Capistrano, you can have different configurations for the different stages. I’ve shown our configuration for the development stage.

As you can see, we’ve set localhost for the server attribute and that means we will use our local machine for the deployment and we don’t need to use virtual-machines or a real server.

As we mentioned before, we have some custom tasks to be run after the default steps of the Capistrano to do the main part of the deployment flow.

We’ve put these custom tasks inside the ./lib/capistrano/tasks/docker.rake file.

I’m going to explain each task briefly and for sure we should have some experience with Docker to find out what’s happening in these tasks exactly. create_network creates backend network and we will use it to run all containers in the same network so they will be able to access each other. create_volumes creates prerequisite Docker volumes. db_setup runs rails db:setup to initialize the database (for sure for a real application you should not execute database-setup on each deployment). And finally start_container will run the application container with the specified environment variables.

Now we’re ready to deploy our application. bundle exec cap development deploy command runs Capistrano deployment flow for the development stage. After running the deployment command you should get the error below:

...

Errno::ECONNREFUSED: Connection refused — connect(2) for 127.0.0.1:22

...

We don’t have any ssh-server on our local machine and that’s the reason we’re getting this error. To install ssh-server and configure the ssh-key pair on the Ubuntu machines you can follow the commands below:

# ------- Install ssh-server, enable, and run its servic -------
sudo apt-get install openssh-server
sudo systemctl enable ssh
sudo systemctl start ssh

# ------- Create a new key-pair -------
ssh-keygen -t rsa -b 4096 -C "Capistrano Local" -f ~/.ssh/capistrano_local

# ------- Add the new private-key for ssh-client -------
ssh-add ~/.ssh/capistrano_local

# ------- Add the new public-key for ssh-server -------
cat ~/.ssh/capistrano_local.pub >> ~/.ssh/authorized_keys

So we will have ssh-server and ssh-client simultaneously on our machine. Now we’re ready to deploy:

bundle exec cap development deploy

When it finished successfully the application container should be running and responsible for the curl request:

curl http://localhost:3000/ping

# ===> {"message":"pong!"}

If we get {"message":"pong!"} response, it means your application container is up and running.

Add Nginx

In our existing structure, the Puma which is running inside the application container receives the requests directly. In this section we’re going to setup an Nginx container and put the application container behind it. For this purpose, we will use jwilder/nginx-proxy image instead of the official nginx image. When a new container is run, nginx-proxy will recognize it and will generate reverse-proxy configs and will reload the Nginx process automatically.

We need to run nginx-proxy only once and it’s better to keep it out of your deployment flow, but for the sake of simplicity, we will use Capistrano to run nginx-proxy container. Also, we need to add a new environment variable(VIRTUAL_HOST) to the application container. The new container will be recognized automatically because of the VIRTUAL_HOST environment variable and it will be used as server-name in the Nginx configurations.

start_nginx checks the running containers and run the nginx-proxy container if it’s not running. We also have two small changes for the start_container task and as you can see, we don’t need port mapping anymore and instead of that, we’re passing VIRTUAL_HOST env to the application container — as we described before.

And we need to update config/deploy.rb file to add the new task to our deployment flow :

Now we need to run Capistrano deployment command again to see our changes:

bundle exec cap development deploy

For testing keep in mind that, the nginx-proxy navigate the requests to our application container if they come for the specified domain which is specified via the VIRTUAL_HOST environment variable. So we need to fake the domain by setting Host header for the curl request:

curl --header "Host: app-domain.test:8080" localhost:8080/ping

# ===> {"message":"pong!"}

Implement zero-downtime with Blue-green deployment (the moment of truth)

The idea is to add a health check to our application container, run the new version of the container without stopping the old one, keep running the old and new versions of the application simultaneously until the new version being responsible for the requests and finally stop the old container.

By adding a healthy check to the application container, we will be able to find out that the container is responsible for the requests or not. An easy way to implement a healthy check for our container is sending curl request to the /ping endpoint — which we implemented in the first section.

So according to our idea, we need to add healthy-check to the start_container task. curl -f localhost:3000/ping will execute every 10 seconds and will update the status of the container. wait_for_container will wait for the new container(zero-downtime-app-new) to be healthy and after that the stop_container will be executed to stop the old container and finally rename_container task will rename the name of the new container from zero-downtime-app-new to zero-downtime-app.

Ensure that you’ve added the new steps to the config/deploy.rb file:

Test zero-downtime with ApacheBench

We want to use ApacheBench tool to test our implementation for the zero-downtime deployment. We will use it to send a lot of requests in the specified time limitation and run the deployment simultaneously. ApacheBench will tell us the count of the Non-2xx responses and this value should be ZERO for us to confirm our solution.

Open two separate terminals on your machine. Run deployment in the first terminal:

bundle exec cap development deploy

and run the ApacheBench in the second one — if this is the first time that you’re running this command, it will take some times to get the ab image from the registry:

docker run --rm --network=backend httpd ab -s5000 -t50 -n1000000 -H "Host: app-domain.test" -c1 http://nginx/ping

After finishing the deployment you can stop the ApacheBench container(Ctrl + C) and it will generate the report:

This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking nginx (be patient)

Server Software:        nginx/1.19.3
Server Hostname:        nginx
Server Port:            80

Document Path:          /ping
Document Length:        19 bytes

Concurrency Level:      1
Time taken for tests:   52.124 seconds
Complete requests:      24705
Failed requests:        0
Total transferred:      13488930 bytes
HTML transferred:       469395 bytes
Requests per second:    473.97 [#/sec] (mean)
Time per request:       2.110 [ms] (mean)
Time per request:       2.110 [ms] (mean, across all concurrent requests)
Transfer rate:          252.72 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       9
Processing:     1    1   0.3      1      13
Waiting:        1    1   0.3      1      13
Total:          1    1   0.3      1      13

Percentage of the requests served within a certain time (ms)
  50%      1
  66%      1
  75%      2
  80%      2
  90%      2
  95%      2
  98%      2
  99%      3
 100%     13 (longest request)

The Failed requests should be 0 and also you should not see Non-2xx responses section at the report.

Todos

• Build application docker-image with Capistrano on the server or CI
• Configure and run nginx-proxy out of the deployment workflow

A Simple Solution for Zero Downtime on Deployments with Docker was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

You spend too much time to deploy your rails project especially on the assets:precompile step, or maybe sometimes you see the following error during the assets precompilation:

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

So for sure, this short article will help you to have 50% improvement only with some minor changes.

If you’re using webpacker in your project, the assets:precompile consists of two stages; we will call these two stages, sprockets and webpacker. Now we want to execute these two stages separately to find out which one is your problem.

Run sprockets stage:

WEBPACKER_PRECOMPILE=false rake assets:precompile

and then run webpacker building stage:

rake webpacker:compile

NOTE: If you want to analyze the performance of the above commands in production mode, pass RAILS_ENV=production as an environment variable to both of them.

If the webpacker is your main problem, keep reading and we are going to investigate and resolve the issue. There is a way to see the details of webpacker compile command, in order to that we should use webpack itself instead of webpacker:

./bin/webpack --progress

NOTE: If you want to run webpackcommand in production mode, you need to pass NODE_ENV=production as an environment variable to this command. For example NODE_ENV=production ./bin/webpack

There should be a time/memory consuming step to generate source-map files.

**% after chunk asset optimization SourceMapDevToolPlugin js/**-**.js generate SourceMap

Source-map files help to map a combined/minified file back to an unbuilt state. If you don’t need source-map files on the production server, you can skip generating these files by setting devtool for the webpack/webpacker. I didn’t find already exist, implemented an environment variable to set this feature, but don’t worry, we will do it manually.

Open environment.js file and add the following two lines before the last line(module.exports = environment; ):

...

const devtool = process.env.DEVTOOL;
if (devtool) environment.config.merge({ devtool });

...

module.exports = environment; // this line already exists

Then test the implemented code:

DEVTOOL=none ./bin/webpack --progress

You should not see the source-map generating step anymore.

Now you will be able to pass DEVTOOL ENV to assets:precompile command:

DEVTOOL=none rake assets:precompile

That’s it. You should have a huge improvement!

How to speed up assets precompile for Ruby on Rails apps was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

In our project, we are using Active Storage as our upload solution, and Amazon S3 for Storage service. Recently because of a requirement in the project, we decided to encrypt files before uploading to S3 and decrypting them after downloading (client-side encryption). Amazon S3 Gem meets this requirement and we can find the documentation through this link.

And the following greate article shows how we can implement it in a transparent way:

Active Storage S3 Client-Side Encryption

This blog post has described what you need, BUT if you want to have a preview for the uploaded files, so you are the one who should read this article.

Preview for Uploaded Files (image, PDF, movie, …)

Unfortunately, Active Storage does not completely support preview for client-side encrypted files, if we use the default previewer path which is created by Active Storage itself, it will give us a URL that refers directly to the encrypted thumbnail image on the S3, and it’s not previewable by an HTML image tag.

NOTE: We should not forget that we need a transparent way that works with any configuration of Active Storage. For example, config.active_storage.service = :local should work properly in your development environment while config.active_storage.service = :amazon works on the production server.

Assume that we have an Attachment model that is associated with on document:

class Attachment < ApplicationRecord
  has_one_attached :document
end

Now, we will go through the following steps:

1. The blog post in the previous section has missed an important line of code that is used in the Active Storage Preview, and we need to change the download method like the following code:

def download(key, &block)
  binary_data = instrument :download, key: key do
    encryption_client.get_object(
      bucket: bucket.name,
      key: key
    ).body.string.force_encoding(Encoding::BINARY)
  end

  yield binary_data if block

  binary_data
end

The bold line is the missing line. If you want to know why we need to add this, check out this line of the Active Storage source code.

2. Then we are going to create a service object in order to process and download preview blob:

module Attachments
  class Preview
    def initialize(attachment)
      @attachment = attachment
    end

    def call
      document_preview = 
        @attachment.document
                   .blob
                   .representation(resize: '200x200')
                   .processed

      if document_preview.is_a?(ActiveStorage::Variant)
        variant = document_preview
      else
        variant = ActiveStorage::Variant.new(
          document_preview.image,
          document_preview.variation
        )
      end
      variant_preview = variant.processed
      ActiveStorage::Blob.service.download(variant_preview.key)
    end
  end
end

3. Now we need to create a controller to serve preview file:

class AttachmentsController < ApplicationController
  def preview
    preview_data = ::Attachments::Preview.new(attachment).call
    send_data(
      preview_data,
      filename: attachment.filename,
      type: 'image/png'
    )
  end

  private

  def attachment
    @attachment ||= ::Attachment.find params[:id]
  end
end

4. And that’s it, now you just need to use the proper route to implemented action in your <img /> tags:

<img src=<%= preview_attachment_path(attachment) %> />

Wrapping Up

If you want to use the Preview feature of the Active Storage you need to know that it generates a thumbnail beside the original file, but when you are using client-side encryption, you can not directly refer to the thumbnail file (because it’s encrypted too). In this case, you need to decrypt the thumbnail file, before previewing. We have explained how you can meet this requirement by creating a custom controller and use it in your image tag instead of the URL which is generated by the Active Storage.

To learn how to create a database view in Ruby on Rails, start by reading the following article:

How to create Database Views in Ruby on Rails?

1. Concatenation methods

Combine ActiveRecord result by the plus(+) operator (or any similar methods):

photo_posts = PhotoPost.all
text_posts = TextPost.all

result = photo_posts + text_posts

There are some drawbacks with this solution:

• Performance issue (Ram based process)
• Really hard to sort and pagination

2. Polymorphic

In this solution, we will create a model (table) with a polymorphic association to PhotoPost and TextPost:

class Post < ApplicationRecord
  belongs_to :postable, polymorphic: true
end

and associate your models (PhotoPost and TextPost) to Post model:

class PhotoPost < ApplicationRecord
  has_one :posts, as: :postable
end

class TextPost < ApplicationRecord
  has_one :posts, as: :postable
end

Note that you need to create Post object manually after creating every PhotoPost or TextPost.

This technique is called multi-table inheritance. There are some gems to simulate MTI for ActiveRecord models (e.g. active_record-acts_as).

Drawbacks with this solution:

• N+1 query problem, Because you can’t join Post model with PhotoPost & TextPost
• Additional model (Post)
• Hard to sort by attributes which are not a member of Post model

3. Database View

Describing Database View is beyond the scope of this article. We just will know how we should use View in Ruby on Rails.

Create a migration to create Database View:

class CreatePostView < ActiveRecord::Migration
  def up
    execute <<-SQL
      CREATE VIEW posts AS
      SELECT
        id AS indentifier,
        title,
        updated_at,
        created_at
      FROM
        photo_posts
      UNION ALL
      SELECT
        id AS identifier,
        title,
        updated_at,
        created_at
      FROM
        text_posts
    SQL
  end

  def down
    execute <<-SQL
      DROP VIEW IF EXISTS posts
    end
  end
end

Rails behaves in the same way with tables & views and you only need to create Post model like the other ones:

class Post < ApplicationRecord
end

Now, All SELECT kind of queries are available by Post model, e.g. Post.first, Post.all, Post.where() .

Wrapping up

In this I’ve listed the advantages and disadvantages of each solution:

Concatenation methods

✓ Easy to read and write
✗ Hard to sort
✗ Hard to paginate
✗ Performance issue (Ram based process)

Polymorphic

✓ Ability to generalize attributes, associations, and methods
✓ Easy pagination
✓ Transparency between the models which have is-a relationship together
✗ Performance issue (N+1 query problem)
✗ Additional model (table)
✗ Hard to sort by attributes which are not a member of Post model

Database View

✓ High performance
✗ Hard to maintain

My Experiences in Live Rails Application was originally published in salamcinema on Medium, where people are continuing the conversation by highlighting and responding to this story.

Hadir is a gem which provides an authorization system like Pundit. With Hadir you separate policy logics and put them in policy classes based on each controller.

Installation

All you need is adding hadir to your Gemfile:

gem 'hadir'

Getting Started

Hadir is focused on policy classes. You need a policy class for each controller and a method into it for each action (It’s possible to use same method for multiple actions).

In the following controller we prevent to update or delete unpublished post, also in delete action we use custom method and messsage:

class Api::V1::PostsController < ActionController::Base
  def update
    post = Post.find(params[:id])
    authorize post
  end

  def delete
    post = Post.find(params[:id])
    authorize post, 'update?', message: 'You are not allowed to delete unpublished post.'
  end

  private

  def current_user
    # retrieve current-user and return it
  end
end

Following policy class will allow updating a post if it is not unpublished:

class Api::V1::PostsPolicy
  attr_reader :user, :record

  def initialize(user, record)
    @user = user
    @record = record
  end

  def update?
    record.published?
  end
end

Hadir makes following assumptions about your policy classes:

• Policy class name is the same as resource part of controller with Policy suffix.
• Policy class has the same namespace as contoller class.
• The first argument is a user. In your controller, Hadir will call the current_user method to retrieve what to send into this class. (null is acceptable)
• The second argument is an object which you want to check its authorization. It can be any object you want.
• As a default behaviour Hadir maps your action to the method with the same name and question mark (?) as a suffix. For example Hadir maps update action to update? method in your policy class. It is also possible to send your desired method name as second argument of authorize method.

Hadir vs Pundit

In Pundit you create a policy class for each object’s class(most of the time a model) and put methods into it for each controller’s action, there are some disadvantages:

• You have only one policy class for different API versions or admin APIs(different controllers) for the same model(class). You will have a messy class!
• You are not able to have policy if you don’t pass a specific object as argument of authorize metho

Hadir was originally published in salamcinema on Medium, where people are continuing the conversation by highlighting and responding to this story.

Why should I use Minio?

As I mentioned before we needed one-time URL of files per each user and I couldn’t use Nginx itself to provide files.

Run Minio using docker

That is really simple to run Minio by its docker image. Install docker and run following commands:‍‍

$ docker pull minio/minio

$ docker run -p 9000:9000 --name minio \
    -v /home/username/_minio_volume:/export \
    minio/minio server /export

After running successfully, you will see Minio credentials (AccessKey, SecretKey), write them down; Or you can pass them by yourself:

$ docker run -d -p 9000:9000 --name minio \
    -v /home/username/_minio_volume:/export \
    -e "MINIO_ACCESS_KEY=XXXXXXXXXXXX" \
    -e "MINIO_SECRET_KEY=XXXXXXXXXXXXX" minio/minio server /export

Open your browser and enter localhost:9000, if every things go right, you will see Minio login page.

Upload file to Minio by GUI

If you see login page, login with your credentials. At first, create a bucketand upload your file to specified bucket:

Create temporary link to your file by clicking on three dot icon in the front of uploaded file; then click on copy icon and finally click on Copy Link button. As you can see in following picture you can change temporary link expiration time:

Using Ruby on Rails to make temporary URL

After uploading a file to Minio you can create temporary link to the file programmatically. Minio APIs are completely compatible with Amazon S3. That means you can use S3 gem. You can use following three steps to create temporary link:

1. Add AWS gem to Gemfile:

gem 'aws-sdk', '~> 2'

2. Creaet initializers/aws.rb file:

Aws.config.update(
  region: 'us-east-1',
  endpoint: ENV['minio_endpoint'],
  force_path_style: true,
  credentials: Aws::Credentials.new(
    ENV['minio_access_key'],
    ENV['minio_secret_key']
  )
)

3. And finally create temporary link:

s3_res = Aws::S3::Resource.new

tmp_url = s3_res.bucket('movies')
                .object('filename')
                .presigned_url(:get, expires_in: 10.days.from_now)

How to configure Carrierwave gem in Ruby on Rails to use Minio

For example in some use-cases you need to upload your files to Minio through carrierwave. In order to this purpose you can use this link.

Running advanced object/file storage server in less than 1 hour with “Minio” was originally published in salamcinema on Medium, where people are continuing the conversation by highlighting and responding to this story.