Real-Time Data Streaming, Routing, and Processing Using Kafka and Snowflake

A top-tier bank’s legacy messaging infrastructure posed multiple challenges in handling growing data volumes – Transaction Data, Customer Data, New Loan Application Requests, KYC Data, etc. Hence, activating any new digital experience using the existing legacy infrastructure meant enabling high volumes of asynchronous data processing. Traditional messaging middleware like Message Queues (MQs), Enterprise Service Buses (ESBs), and Extract, Transform and Load (ETL) tools were unable to provide the necessary support that modern applications demand.

Modern Applications Require Asynchronous, Heterogeneous Data Processing

What is Asynchronous Data Processing?

Asynchronous processing allows the system to handle multiple loan applications simultaneously without waiting for each application to complete. This means that while one application is being reviewed, others can continue to be processed in parallel.

For example, when a borrower applies for a mortgage loan through an online lending platform, the backend must be capable of collecting required documents and information, such as income statements, tax returns, credit reports, property details, and employment history.

When the borrower submits their application, the system immediately acknowledges receipt and starts the process. Meanwhile, in the background, the system also asynchronously verifies employment history, orders a credit report, and assesses property value.

Why Enable Data Streaming, Routing, and Processing Using APIs?

With the implementation of a Digital API Hub, the Legacy Messaging Middleware gets integrated with modern event streaming automation tools. It can then be taken enterprise-wide to enable new services or functionalities using the existing data.

How are Reusable Microservices Built Using a Modern Messaging layer?

The new messaging layer helps create reusable components from existing topics or data. Hence, launching any new digital service or feature can consume existing topics or data. A topic here implies code that is inside of a Terraform module which can be reused in multiple places throughout an application.

Why Choose Kafka and Snowflake for Real-Time Data Streaming

Snowflake as a data warehousing architecture and Kafka as a platform was chosen to automate different data stream lanes. Our developers were able to used Snowflake to enable event-driven consumption using Snowpipe. By integrating this cloud-based system, we were able to provide easy access to more cloud-based applications for different banking processes and teams.

• We set up a Java application for data producing teams to scrape an API and integrating it with the data routing platform.
• Using Kafka as a buffer between data producers and Snowflake allowed for decoupling of ingestion and processing layers, providing flexibility and resilience.
• Information on different topics is then pushed into further processing for sending our event-driven notifications.
• We also set up different event-driven data streams that achieves close to real-time fraud detection, transaction monitoring, and risk analysis.