Machine Learning Recommender Systems: 4 Key Insights From apply(recsys)
Recommender systems are a widespread use case for machine learning, but despite their ubiquity, approaches and best practices for building them are still developing, especially for real-time ML. Tecton’s apply(recsys) conference brought together leaders in the space to explore the practical challenges in serving great recommendations.
Whether they want to surface videos, products, job opportunities, or anything else, teams building recommender systems (recsys) must combine an understanding of ML theory with the realities of engineering performant, maintainable systems. In this post, we’ll summarize four major takeaways that any team interested in recsys should keep in mind.
1. The gap between theory and practice is wide
Many speakers at apply(recsys) brought up the gap between how recommendations work in theory and how we can actually build recommender systems. From data ingestion to the way recommendations are served to the user, every step of the journey offers its own challenges.
In the panel discussion “Lessons Learned: The Journey to Operationalizing Recommender Systems,” Jacopo Tagliabue, MLSys Professor at NYU, suggested you check if you actually have the data to make good recommendations before you start because it turns out many teams don’t. Some examples include tracking user interaction and whether you have enough metadata about the books, movies, or whatever you’re recommending. If you don’t have enough data to ask a recsys question, he suggested getting your data story together and setting up a stronger feedback loop first rather than moving forward with trying to build a recsys with poor data.
It’s also important to ensure your model is deployable and useful in practice, Agnes van Belle, Team Lead of Data Science at HeyJobs, warned during the panel. She said that it’s easy to overemphasize offline experimentation. For instance, teams can spend a long time building a model that performs great offline but then realize that deploying that model would be too resource intensive. In other cases, she has seen teams forget to incorporate critical business logic the online model will have to contend with, such as paid ads. She recommends getting a model online quickly to ensure good alignment: “Focus on getting the whole flow of the end-to-end pipeline, and at that point, just deploy a simple model. Then you can iterate over time with regular online testing.”
Krystal Zeng, Machine Learning Engineer at Cookpad, advised that recommendations need to be considered in their product context. Her team had developed an offline model that showed great performance, and they expected it to drive a lot of user interaction. However, once the model was deployed, it didn’t perform as they had hoped. After investigating, it turned out users didn’t open the feature they had developed to serve those recommendations. “There was a design problem that we didn’t know about,” she explained. “Offline, there were no design [considerations], and we ran with the assumption that all the users who wanted to interact, would interact.” Marc Lindner, co-founder of eezylife, agreed: “It sounds simple, but knowing if your UX works is probably one of the hardest problems you have to deal with.”
2. Real-time recommendations open up new possibilities—and building online models to serve them is getting easier
Many speakers spoke about the value of making recommendations based on real-time data, as well as the growing platform support for deploying online models. While pre-computed batch recommendations can be high quality and are simple and fast to serve, they come with notable limits and drawbacks, leading to growing interest in real-time recommendations. Mike Del Balso, CEO of Tecton, said, “Using real-time data adds complexity, but it can also have a disproportionate improvement in the impact of your application.”
A major limitation of batch recommendations is that they have to be computed ahead of time, so they can’t account for context and user behavior at any given moment—for example, as they’re browsing a shopping site for kitchen gadgets. Users may have different needs based on the device they’re using, time of day, or previous behaviors like what items they put in their cart. Youlong Cheng, Engineering Leader at ByteDance, pointed out the importance of online recommendations for TikTok: “We prefer instant feedback from users. If you can capture the user’s immediate interest faster and change your recommendation result quickly, that can improve the user’s experience.“ (Check out Youlong’s full talk “Monolith: Real-Time Recommendation System With Collisionless Embedding Table,” where he shared the technical details of ByteDance’s recent work in building a system for large-scale, online item recommendations.)
In some cases, batch recommendations may not even be possible. Jacopo pointed out that most e-commerce sites only see users rarely, and those who return do so at an average rate of less than twice a year: “You have no idea what this person wants or does ahead of time. If your only strategy relies on having user history, you will fail in the vast majority of cases. In that case, the only thing you can do is go online [aka real time].”
Luckily, it’s getting easier to build out real-time recommendation systems. In their workshop “Choosing Feast or Tecton for Your RecSys Architecture,” Jake Noble, Software Engineer at Tecton, and Danny Chiao, Engineering Lead at Tecton, explained how feature stores can simplify real-time recommendations. For example, a feature store will help scale your data service layer, which is important “because for every request that you get from your front end, that will fan out to hundreds of items that you’re going to end up serving. Your data service layer needs to be able to handle a hundred times whatever your front-end service sees.”
3. Unbundling recommenders provides more power and flexibility
Some speakers recommended creating individual modules for each step of your recommender system—e.g., components like a candidate generator, a feature store, a scoring service, etc. As recommendation systems become more advanced, componentization can provide great benefits because isolation provided by this approach makes it easier to collaborate across teams without issues. It also makes experimentation easier, allowing for faster iteration.
And if you find your business has particular needs or challenges, it’s easier to modify the architecture. As an example, in his talk “Crawl, Walk, Run: A Practical Intro to Applied Recommender Systems,” Mike described an ad-recommendation system with a large number of candidates. You can easily implement a tiered approach by duplicating components—first by sending ads through a lightweight ranking to filter out the top options, then running a heavier ranking on the remaining candidates.
However, a componentized architecture can add significant overhead. “You want to have tooling that allows you to spend as little time as possible on tooling, frankly,” Mike said.
4. Consider when it makes sense to build a recommender system in-house—or when it makes sense buy one instead
A frequent question from attendees is when it makes sense to build a recommender system. Every business has its own resources and needs, but two major questions arise:
- Should I buy an end-to-end recommender system or build it in-house?
- At what point does it make sense to build a system?
In the panel, Krystal said that building your own in-house solution means better control of the system. She shared that, at Cookpad, building their recsys in-house provided better visibility into what their model was doing, made it easier to debug, and gave them opportunities to fine-tune their stack. However, she also acknowledged that having a well-established infrastructure helped Cookpad get their recsys up and running quickly. In short, “if you are a company starting out that doesn’t have a lot of components working, a well-managed third party [solution] would make it quicker and easier to plug in and test things out,” Krystal said.
Once your infrastructure is solid, though, you don’t need a large team to build a recommender system. Krystal said their team is just five researchers and two infrastructure engineers. Echoing Agnes’ advice, Krystal said, “We went for a very simple and direct solution. We test things quickly online so that we can validate our features.” Marc Lindner, Co-founder and CPO at eezylife, said eezylife also has a fairly small team, with three researchers and six people working on the data pipeline. If you start with simple infrastructure, you can then build up to more complicated architectures over time.
Finally, Jacopo gave a simple check for whether to buy or build: “If your core business depends on how good your recommendations are, now or in the near future, build it yourself. If not, just buy it and save a lot of pain.” He emphasized that if you buy, it’s easier to start with success and build with confidence instead of dealing with the upfront cost of building a system and not knowing if it will succeed.
Next steps
These four takeaways are a high-level overview of some major discussion topics that came up at apply(recsys). For a deeper dive into the topics, we recommend watching the full apply(recsys) sessions, available on demand.