2021 was a big year for Tecton! Our mission is to bring ML intelligence to every production application, which starts with the world’s best enterprise feature store to power machine learning. We made huge strides towards this mission:​​ demand for feature stores skyrocketed, our list of customers increased more than 5x, and the team shipped a number of new mission-critical product features. As the number of customers ramped up, the demands on our engineering team grew.

At the same time, the open source feature store, Feast, exploded in popularity (over 400,000 installations and counting).  Back in 2020, we realized that we were on the same mission as the Feast team, so we decided to join forces and Willem Pienaar joined Tecton as a Principal Engineer. Feast and Tecton have a common vision for feature stores and collectively provide users with both an open source and managed enterprise feature store to choose from.

Our engineering team has grown steadily to meet these demands over the past 2 years. We’ve come a long way from the team being a few engineers working in two tiny offices connected by an always-on Zoom call. As more Tectonauts joined the team from different backgrounds and areas of expertise, we were pushed to rethink its structure to keep up with Tecton’s growth. Supporting both an enterprise and open source feature store added a new set of parameters to the challenge.

Principles of team design

We started the team design process with a few key principles:

• Teams should have clear missions and ownership. Projects come and go, but missions are more resilient over longer time horizons.
• Teams should be structured to foster both team and career growth. It’s critical for teams to be able to grow while providing paths for engineers to grow in their careers.
• Build strong team identities. Building a team’s culture and identity takes some time and we try to cultivate them starting from the day it’s formed. Projects don’t always cleanly fit within the charter of a single team, so we encourage teams to build systems that welcome external contributors. This also promotes clean interfaces with detailed documentation.
• Build a team of passionate experts. Engineers with enthusiasm or experience in a particular area should be able to join any team regardless of where they’re located.

We settled on an organizational structure that combined vertical and horizontal teams on different missions.

Product Engineering, Platform Engineering, and DevOps are fairly typical among growing companies, but we also created Feast and Spike teams focusing on related, but different missions.

Spike Team

The name “Spike Team” sounds cool and it’s actually a real thing! The Spike Team’s mission is to deliver on high-impact projects that are both strategic and ambiguous in a short timeframe. In other words, the Spike Team is the home of high-risk, high-reward projects.

Spike projects often span the entire stack and introduce new technologies. The team is led by Mike, who spent years building search and indexing at Google before joining Tecton as one of our earliest engineers. You would have a hard time finding a file in our codebase that doesn’t have Mike in its commit history. The key to success for Spike team engineers is the ability to make progress with a high degree of ambiguity and technical uncertainty.

The Spike team operates very differently compared to other teams. The team emphasizes prototyping and beta versions while failing fast as needed. Projects that see early success will often be built in focused, intensive efforts spanning weeks or months, after which the project will graduate to general availability with the help of the Product or Platform teams. The Spike Team recently shipped a series of improvements that made it possible for Tecton’s online serving infrastructure to serve more than 100,000 features per second from DynamoDB, a 10-20x improvement over previous benchmarks. Stay tuned for the next Spike project!

Product Engineering

Product Engineering’s mission is to build a fantastic experience for Tecton’s users from sign-up through operational usage. This covers all user-facing surfaces within the product: APIs, Web UI, Tecton CLI, and Python SDK. This team is the natural destination for product-minded engineers who are passionate about UX and deeply empathize with users. The team is a combination of engineers experienced in both applied ML (like Brian who built deep learning systems at Determined AI) and data infrastructure (like Amanda who worked on caching and performance at Rubrik).

One of Product Engineering’s challenges is building for a diverse set of users with ML teams, which include ML engineers, data engineers, and data scientists. Many of our customers like Atlassian are rooted with an ML infrastructure team that drives the adoption of Tecton internally across teams building ML applications. It’s critical for these different personas to have a seamless experience using Tecton so we formed multiple teams within Product Engineering, each focused on a mission targeting a different persona. One team is focused on building a framework that enables feature engineering for advanced ML use cases, and the other focuses on key features needed to adopt Tecton in large organizations.

Engineers work with PMs (Product Managers) daily and, for some projects, even operate in an informal PM capacity. Some of Tecton’s most highly requested functionality was built in projects driven by engineers from product requirements through to launch. For example, Jake, who previously helped build YouTube Recommendations, drove array features for embedding from end to end. Tecton is a highly technical product for technical users, so PMs appreciate a high level of collaboration with engineers and lean heavily on their product and UX intuition.

Feast

Feast is our open-source feature store that’s growing quickly (2700 GitHub stars and counting!). We invest heavily in pushing Feast forward and many of Tecton’s innovations often start within the Feast community. The team works closely with contributors from Shopify, Twitter, and Gojek and advanced ML teams at companies like Robinhood that choose Feast as their feature store of choice.

We joined forces with Willem and Feast since it shared Tecton’s vision for feature stores. Internally, the Feast and Product Engineering teams work side-by-side to design standardized API boundaries, shared components, and workflows that both Tecton and Feast’s users will benefit from.

Feast is the team for engineers who love collaborating and building in the open. The team engages and learns from a vibrant and diverse community. Teams from both large and small companies have joined the community to learn from others and be part of building the canonical feature store stack. Some of Feast’s most impactful new features were developed and contributed by the community, such as the Feast-Spark connector built by Microsoft Azure or Redis integration built by Redis. If you’re interested in getting involved in the Feast community, join a community call or join the conversation on Slack.

Platform Engineering & DevOps

Platform Engineering and DevOps teams are on a mission to build a robust storage and compute foundation for a reliable, high-performance feature store. Building an internal feature store is a massive undertaking, but building a feature store as a managed SaaS service adds new dimensions of complexity. Tecton is opinionated about the technologies that are best suited to solve the data challenges of Operational ML. Instead of building our own proprietary compute or storage engine,  Tecton integrates with a few best-in-class compute engines as well as online and offline stores, each with unique performance and scaling characteristics. Offering this choice is crucial because trying to fit the sheer diversity of Operational ML use cases into a “one size fits all” offering is guaranteed to fail.

The Platform Engineering & DevOps teams attract engineers who care deeply about scalability, performance, and reliability. Most of the team has built infrastructure at previous companies, including Emma, who built cloud monitoring infrastructure at Salesforce, and Daryl, who led the platform team at SingleStore. The team is constantly breaking its previous records in every dimension of Tecton’s scalability. For example, ultra-low latency streaming bridged the gap from near real-time streaming (<1 minute from steam event to ready-to-serve freshness) to true real-time streaming (<5 seconds event-to-serving freshness).

There’s an engineering heuristic that says systems can only survive 1-2 orders of magnitude growth before needing to be re-engineered. This is especially true of the systems built by Platform Engineering. As Tecton’s customer list grows, new challenges have appeared at every corner of our infrastructure.

It’s all about the team

Mike Saparov, our VP of Engineering, always says “it’s all about the team”. This tenet drives our priorities as we grow the team. Here are a few things we feel strongly about:

Individual growth tracks

One of the key principles is “teams must be structured to leave room for both team and career growth”. For engineers, this usually means growing along either the IC (individual contributor) or manager track.

On the IC track, we prioritize team structures that leave room for ICs at any seniority all the way up to staff and beyond. Staff engineers are responsible for solving business-level problems and can be responsible for setting technical direction, mentorship and sponsorship, providing engineering perspective, and exploration (this list is from one my favorite resources, StaffEng: Leadership beyond the management track). We believe this level of impact should be achievable on every engineering team.

On the management track, we believe engineers who are capable and motivated to try out management in their career should have the opportunity to do so as a tech lead manager of a small team. We always prefer promoting from within to making an external hire. This sometimes produces thriving engineering managers, while others may decide it’s not for them and return to being an IC.

Stay close to the user

Ultimately, we’re building Tecton and Feast for the users. Users come in different shapes and sizes so it’s important for engineers, regardless of the team they’re on, to develop empathy for users. To build this empathy, we’ve created an Engineering POC (Point of Contact) program that pairs engineers with customer success to build relationships with specific customers and go deep into their ML use cases as they build on Tecton.

Engineering POCs tend to gain greater empathy from working with customers than with any form of product testing and dogfooding. It also provides the opportunity to go deep into different sectors since Tecton’s diverse customer list includes software companies, gaming, e-commerce, fin-tech, insurance, and many other sectors.

Distributed teams with hubs

We moved toward a distributed model with hubs in San Francisco and New York during the pandemic. Offices in these cities are available to all Tectonauts. We also have a growing presence in Seattle, LA, Chicago, Honolulu, Knoxville, and even Mexico City.

We’ve been intentional about keeping teams distributed to maximize choice for new hires joining the company — the last thing we want is a scenario where a storage expert in Denver can’t join a team because it’s based in New York. The Product Engineering team, for example, is composed of engineers and managers on both the east and west coast. As we settle into the post-pandemic normal, there are undeniable benefits to in-person meetings so we try to do a company-wide retreat at least once per year along with smaller team-level offsites throughout the year (no more axe throwing, please).

What’s next?

2022 is an ambitious year for Tecton. We’ll be growing the engineering team by 2-3x and shipping several key product features and milestones (can’t say more right now :)). We’re hiring engineers across the board: frontend, full stack, infrastructure, open source, and DevOps, along with engineering managers and product managers. The senior technical team is in search of staff and principal engineers to tackle a number of mission-critical product and infrastructure challenges this year.

We’d love to hear from you if you’re interested in building Tecton with us! Check out our careers page for more details. We have more challenges that need solving than engineers to solve them :).