Lambda Architecture in Google & Azure Could

Lambda Architecture Definition

Lambda Architecture is a data-processing architecture designed to handle massive quantities of data by taking advantage of both batch stream-processing methods to design a robust, scalable and fault-tolerance (human and machine) big data systems.

Lamba Architecture tries tries also balancing between the latency & Accuracy.

Lambda Architecture Layers
Master Layery
Serving Layer
Speed Layer

Lambda Architecture Properties:

  • A paradigm for Big Data
  • In data processing for balance on throughput , latency, fault-tolerance and scalable.
  • For modern data warehouse

Applying the Lambda Architecture with Spark, Kafka, and Cassandra

The toolings are the following:

  • Spark Data Frame & Spark SQL in addition to Spark’s Data Source API to load, store and manipulate data.
  • Spark Streaming & Spark-Kafka Integration techniques -> for reliability and speed
  • Develop a Kafka Data Producer -> to simulate the real-time data stream feed into streaming application.
  • Stateful Spark Streaming Application -> to preserve global state and use memory efficiently with approximate algorithms.
  • Errors & Code updates -> when we build a stateful Spark streaming application and a production application isn’t complete without the ability to handle errors and code updates.
  • Persist Data to Cassandra & HDFS -> for working with the scalable NoSQL database and persist the data to Cassandra and HDFS.
Note

Your Text Here

Lambda Architecture on Azure, Google and AWS

AzureGCPAWS

Related links

References:

How to build a Big Data Pipeline

One thought on “Lambda Architecture in Google & Azure Could

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: