A year in perspective

It’s hard to be able to condense a full year of hard work in a few paragraphs but there’s a reason why people say that a picture is worth a thousand words:

This is a video of what happened in the Unsplash API codebase during the last year.

It’s naive to talk about the future without taking a step back and looking at what we’ve accomplished the past 365 days of building the Unsplash API.

The API received ~19.5 billion requests during 2019. That means we had almost two times more traffic than in 2018.

The team merged 731 PRs in just the API repository, excluding all of the satellite apps, like the frontend, data pipeline, CDNs, and helper services.

All of this was supported by a team of three people and a Rails application that day by day becomes less Rails and more pure Ruby.

A lot of people make Unsplash work everyday but this is a small recap of what a team of three can do.

Monitor every heartbeat

There’s a lot happening within Unsplash, a lot of services, databases and different teams. Our current weapon of choice for monitoring is DataDog. We not only have agents in most of our runtimes but we also have specific metrics we use through our monitors.

Monitors exist within DataDog as a way to identify outliers, anomalies or just things passing a threshold. Mostly we use them to check response times across our architecture but they can serve other purposes like checking if some fail safes are in place or if our experiments are running smoothly.

Experiments did you say? Yes. We run performance experiments when we want to try refactors while ensuring that performance is not going to be affected. All of that data goes to DataDog.

Recaptcha

We love bots, we have some doing tasks for us and a special one we use for chatops. But if we learned something from Terminator it’s that eventually they go rogue.

2019 saw a rise in spam accounts and finally we took the decision to force ReCaptcha on our Join page

This was not an easy decision but as bots become smarter and smarter, we decided to add that filter to keep the content quality high.

There are also some internal tools that detect bots on the platform but that’s a post for another day.

Penumbra

Search is a big and complicated problem to solve and for the past year we’ve been migrating to solutions more close to the metal than before.

To allow better interactions and more responsibility in our code we built Penumbra, a Ruby library to interact with our Elastic cluster, handle mappings, data migrations and search algorithms.

We feel that taking ownership of the core features of our app allows us to take it to the next level. This is obviously a long path that involves a lot of learning but it’s the only one that allow us to squeeze our servers to the fullest and take control of every part of the experience.

Eventually Penumbra will be released as open source.

Autobahn

Popularity comes to the cost of DOS, crawlers and abusive api applications so certain things need to be put in place to ensure that the uptime of the public API is met.

Autobahn is a small middleware that allows us to ban and block IPs based on their behaviour. Since speed is key the backend of it is Redis.

Dynamically stopping abusive IPs was was key to keeping our response times low in 2019.

Goodbye Cervantes, hello Lisboa

Taking ownership of our search algorithms allowed us to start experimenting with different iterations of them. Everything happened behind the curtains and via different experiment flags.

The initial set of algorithms after migrating to Penumbra was aptly called Cervantes. Its main goal was to keep all of the old features without affecting conversion rates, so strictly speaking it did nothing new but did everything the old version did.

We tweak the algorithms quite often but from time to time when the time is right and the changes are big enough we “bump” the name.

Lisboa was the next version of search from which involved a lot of work of one of our teammates: Roberta. Due to that extreme commitment to the craft we named it because of Adriana Lisboa which shares, among other things, the same nationality as her.

This version significantly improved multi-keyword searches versus the prior versions multi keyword searches from the prior version.

Color filtering

As part of our ongoing integrations with the Data team we collaborated to add color filtering to search.

An AI will “look” at the photos and check for color coverage to return color information. All of this was later on added to our search cluster so now we can use the new specific color information to allow filtering among the photos.

Due to the vast amount of fantastic black and white photography and the limitations of AI, the black and white filtering was more challenging to implement and get right.

This colorful version of search is an example of the many experiments we run on regular basis to test new features.

Specialized workers

As one would expect a lot of the work that happens on our servers happens asynchronously and we have a lot of it.

Due to that and with the spirit of DRY we are migrating some of our workers to enqueue jobs as we do normally but process them in bulk.

This MapReduce approach allows us to reduce our execution time for things that need to happen in the background but can wait so the load in our databases/services is not that big.

To homogenize our asynchronous work we started migrating our current cron jobs to be part of the background jobs queue so we get better retry capabilities and cleaner code overall.

Safer together

I don’t want to end this year in perspective without saying thanks to the security researchers that use their time to check our security practices, send emails and communicate with us.

Their love to the craft and help is invaluable to make Unsplash the website it is. Some of them are found here https://unsplash.com/security and we are truly thankful for their support.

Future

There’s so much stuff already for the 2020 edition. I’d love to tell you all it but you’ll have to wait.

As a sneak peek we are shipping a new statistics cluster which is being fed by our first Go service that handles the ingestion and transformation of data.

A lot is happening on API right now and we’ll have a lot to share this year while we push the greatest photo library to the next level.