Fighting Fraud in Real-Time: North’s Journey to Smarter, Faster ML
Every millisecond counts in fraud detection. ML-driven fraud detection systems must evaluate millions of transactions in real time, making split-second decisions that balance customer experience with risk. The technical challenge is immense: processing massive data streams while maintaining ultra-low latency at scale.
North (formerly North American Bancard), a leading payment processor, solved this challenge by building an advanced fraud detection system engineered for real-time processing at scale. Ben Orkin, VP of Engineering – MLOps at North, recently shared how his team architected a platform that delivers consistent, low-latency fraud decisions while enabling quick model iteration.
In this blog post, we’ll explore the key technical insights from North’s journey—from solving sub-second latency requirements to making architectural decisions that support real-time processing at scale. Their experience offers valuable lessons for any engineering team building ML systems where speed and accuracy can’t be compromised.
Inside a Real-Time Fraud Decision
When a customer swipes a card or enters their payment details online, the entire chain—customer, merchant, processor—expects a near-instant decision. Will this transaction be approved or declined? Is it legitimate or fraudulent? Beneath the surface lie complex considerations: the card’s recent usage history, geolocation signals, device intelligence data, and subtle indicators of risk.
As Ben puts it, “You’ve got less than a second to do everything you need to do. People are not going to wait, and a lot of that is making sure that not only is the data right but that it’s done in a timely manner.”
This sub-second window creates a complex balancing act. Block too many legitimate transactions, and you frustrate customers and damage merchant relationships. Let too many fraudulent ones through, and you expose the business to significant losses. For North’s engineering team, this meant building systems that could not only process vast amounts of contextual data quickly but also adapt rapidly as new fraud patterns emerge.
Moving from Rules to ML Models
Historically, fraud prevention relied heavily on static rules: “If transaction X meets condition Y, then flag it.” But as criminals became more sophisticated, rules alone could not keep up. Real-time machine learning models became essential, as they could dynamically learn patterns, factor in new data signals, and adapt to emerging fraud vectors.
North recognized early that to use ML effectively, they needed to contextualize transactions. “When I look at a financial transaction,” Ben explains, “I can’t necessarily tell you what that customer did two hours ago.” To provide richer context, you need access to historical profiles, geolocation signals, email risk scoring, and device fingerprints—often from external data sources. “It’s really a matter of being able to generate real-time profiling for individual data points and looking at the context of that transaction, as well as augmenting that financial transaction with any third-party data you might want to bring in.”
Designing for Real-Time Data Complexity
Building a real-time fraud detection system isn’t just about plugging in a model. It’s about ensuring that all data pipelines, feature computations, and model inference services operate at sub-second latencies. High throughput and low latency mean you have to carefully orchestrate streaming and batch data sources, preprocess features, and serve them at lightning speed.
The technical challenge is compounded by data quality and security concerns. You must ensure data freshness, unify disparate data formats, and secure sensitive cardholder details. North’s solution required re-architecting their backend to break down monolithic systems into smaller services, each optimized for speed and reliability.
“Anything we build ties tightly to the authorization and settlement rails,” says Ben. “We make sure that processes we build are close to the payment systems in terms of geography and latency, and that they’re real-time.”
The Need for a Flexible, High-Performance Feature Platform
Building robust fraud detection models depends on getting the right features—signals derived from raw data that help the model distinguish between fraud and legitimate transactions. Historically, feature engineering has been a bottleneck. Without a proper system, data scientists are forced to reinvent features, apply transformations inconsistently, or struggle to backfill historical data for training.
“You can say, ‘I have a Redis cache where I can pull features from,’” Ben notes. “But when you start pulling out the individual components, the questions become: who’s my customer for this feature store? If it’s data scientists, I don’t want them to have to learn a new language or build everything from scratch.” North needed a solution that was data scientist-friendly, supported version control and CI/CD, and provided robust backfill capabilities.
North turned to Tecton, a platform designed to streamline the creation, management, and serving of features in production. “As we started looking at various feature stores,” says Ben, “One thing really resonated with us—we need the ability to speak the language that our data scientists speak.” Tecton checked these boxes by enabling Python, SQL, and PySpark support, and providing the necessary integration for backfills and iterative development.
Accelerating Iteration and Time-to-Market
Before implementing a feature store, North’s data science and ML engineering teams faced long cycle times when creating or refining models. Without a standardized feature platform, “it was months to iterate,” Ben recalls. After adopting Tecton, that time shrank dramatically. “We brought our time down from months to weeks, shooting for days as we bring in more automation in our CI/CD pipeline and our MLOps pipeline.”
This acceleration doesn’t just improve engineering efficiency; it drives better outcomes. Faster iteration cycles mean North’s data scientists can experiment with new features, incorporate new data sources, or adjust model strategies in response to evolving fraud patterns. They can ask questions like “What if I had data point X or Y—would it improve the model?” and quickly find out. This level of agility leads to more robust models that catch more fraud while minimizing false declines.
Balancing Accuracy, Latency, and Scalability
Building a model that catches fraud is one thing; ensuring it scales economically and remains stable under peak loads is another. North designed their system so it could scale horizontally in the cloud, ensuring that the heavier midday transaction load is handled seamlessly while consuming fewer resources during off-peak hours.
“Pretty much everything is in the cloud,” Ben says. “We had the opportunity to deploy entirely in the cloud and have Tecton do a lot of the compute on their side, while we maintain data retention. This approach helps ensure that all components—from real-time scoring services to feature transformations—are managed efficiently.”
North also implemented circuit breakers and redundancy measures to handle model or feature store outages. After all, a merchant cannot afford downtime. If a model or feature service fails, the system must gracefully degrade, ensuring transactions don’t grind to a halt. “As soon as someone says ‘I’m going to inject a model into a payment authorization workflow,’ the first question is: what happens if it fails?” Ben notes. “You can’t allow downtime; it’s a bad day for everybody.”
Beyond Payment Fraud: Future Directions
With a robust real-time fraud detection system in place, North is looking beyond just authorization fraud. They plan to apply similar ML-driven insights to areas like anti-money laundering (AML) and merchant onboarding. By leveraging Tecton and its real-time architecture, North can score new merchants, validate financial statements, and detect account takeovers, expanding their fraud prevention capabilities and driving value for customers.
They also foresee more specialized, industry-focused models. Instead of a one-size-fits-all global fraud model, North can create niche models tailored to merchant verticals or transaction profiles. “We really want to start breaking the model down,” Ben explains, “looking at opportunities to say: are there industry-specific models? How do we specialize models so that they’re more adaptive either to industry or whatever the data scientists decide is best?”
Key Engineering Takeaways
For teams at financial institutions, e-commerce platforms, or any business fighting fraud, North’s experience offers several lessons:
- Real-Time is Non-Negotiable: Latency is the enemy. A successful fraud platform must deliver sub-second decisions.
- Context Matters: Pure transaction data is too limited. Incorporate historical, third-party, and streaming data signals to paint a richer risk picture.
- Feature Engineering Efficiency: Adopting Tecton unlocked faster model iteration, ensured consistent feature definitions, and simplified backfilling historical data.
- Agility Over Perfection: Instead of deploying monolithic, rigid solutions, choose modular approaches that allow data scientists to experiment, adapt, and improve over time.
- Scalability and Reliability: Cloud-based, containerized systems with proper CI/CD, redundancy, and circuit breakers ensure that your platform scales with your business—and doesn’t fail at critical moments.
Want to dive deeper into building ML-powered fraud detection systems? Watch the full conversation with Ben to hear more technical details about North’s infrastructure, their approach to feature engineering, and lessons learned along the way.