How one new feature in Trypema turned into a major version release

I'm happy to announce that the Trypema rate limiter now has a cleaner API. It all started when I wanted to include a new feature into Celeris (the platform I've been rambling about for the past year). While we have our normal rate limiter working perfectly, there are still some places where it isn't perfect. For instance, we have a per-month rate limiter for free accounts, if we have an issue with redis (for instance, a cluster outage), we wouldn't want to clear all the data within the redis instance or rely on stale data which is what prompted the features that would be discussed here.

For a better background on the problem, I would explain how the current rate limiter works. Most rate limits we have on the platform work with a sliding window algorithm with the sliding window being 6 seconds. If we lose all the traffic data in redis, the cost of this won't be high. It just means we are resetting all rate limits.

But there is an issue with our longer termed rate limits. Namely, the monthly message limit for free accounts. While our philosophy is to store ephemeral data in redis, we need a mechanism to account for missing data for this longer termed rate limits. If the window is 1 month, we can't just ignore what has happened in the past one month and reset the limit as we did the other limits.

Solution 1: Just check if we have anything in redis for that key

This solution is simple, when we are about to create a server for any channel involving the free account, we check the rate limiter for that key and see if we already have something in it. This is why I introduced the get, and get_estimate functions in the providers (local, redis and hybrid). While this seems to solve the problem, it seems to ignore a critical issue, it assumed when we have an issue with our redis cluster, we delete all the data within it. Given that redis can now store data to disk, an extended period of outage would still mean that we can retrieve data from the stale cache, and since it would still be in the window, we would interpret it as valid. This brings us to the second solution, more like something to build on top of this.

ps: Even though this method isn't what I later went with, I chose to still include it in new release of Trypema since someone might find it useful.

Next layer: priming redis with the data we want

Our long term usage data is not stored in redis, but in a separate database. We can't just use the redis client to get the data, because we don't know if the data is stale or not. Instead, when we are about to create the channel server, we would prime the system. It is very simple, we just get the usage data from our more durable datastore for the past month, then set the free account's usage to that value. Hence, why the set functions was introduced in the providers.

This didn't fix race conditions though, since we sync our usage data into the persistent store every 30 seconds to 1 minute, it means the priming logic can run for a channel server while another has already done that and is increasing the monthly usage in redis. If the interleving priming logic sets the value, we basically lose everything in between. So, I introduced the set_if_smaller function to the providers to set the value only if the value in redis is smaller than the one I'm about to set into it. Since other people can have different conditions, it was updated to a generic set_if function that takes in the condition for setting the value. Which can be Eq, Lt, Gt, Ne, or Always.

And voila, that was the solution in my case.

The next things

While I was at it, I also added a few more features to the set function and called it set_if_preserve_history. This is because if you want to change the value of the window (particularly reducing it, or increasing without resetting the window). You might not want to reset the whole window. So, the previous set_if, function was adjusted to do a HistoryUpdateMode::Replace, which means it resets the whole history and sets the value in a single bucket set to now. set_if_preserve_history on the other hand respects the previous history that already exists and you can tell it to either preserve the latest updates (it will mutate the farthest buckets in the windows first then work its way to the latest), or preserve the oldest (it would start its mutation from newest buckets to oldest buckets).

While these didn't warrant a major version bump, they are still worth mentioning. There were naming conventions that I wasn't really happy with and the bigger issue was that when you create a RateLimiter in the v1, it basically starts all the providers. This is not ideal, since you might want to start only the ones you need. So, I made the providers (local, redis and hybrid) separate with fairly consistent builder patterns. There were other changes though, you can see the full changelog here.

Closing notes

I thank thee for partaking in the reading of this my diary (or logs). I hope you enjoyed it. Adios!

Read the v2 documentation, the API reference, or the source.