Architecture

Lambda Architecture in Google & Azure Could

Lambda Architecture Definition

Lambda Architecture is a data-processing architecture designed to handle massive quantities of data by taking advantage of both batch stream-processing methods to design a robust, scalable and fault-tolerance (human and machine) big data systems.

Lamba Architecture tries tries also balancing between the latency & Accuracy.

Lambda Architecture Layers
Master Layery
Serving Layer
Speed Layer

Lambda Architecture Properties:

  • A paradigm for Big Data
  • In data processing for balance on throughput , latency, fault-tolerance and scalable.
  • For modern data warehouse

Applying the Lambda Architecture with Spark, Kafka, and Cassandra

The toolings are the following:

  • Spark Data Frame & Spark SQL in addition to Spark’s Data Source API to load, store and manipulate data.
  • Spark Streaming & Spark-Kafka Integration techniques -> for reliability and speed
  • Develop a Kafka Data Producer -> to simulate the real-time data stream feed into streaming application.
  • Stateful Spark Streaming Application -> to preserve global state and use memory efficiently with approximate algorithms.
  • Errors & Code updates -> when we build a stateful Spark streaming application and a production application isn’t complete without the ability to handle errors and code updates.
  • Persist Data to Cassandra & HDFS -> for working with the scalable NoSQL database and persist the data to Cassandra and HDFS.
Note

Your Text Here

Lambda Architecture on Azure, Google and AWS

AzureGCPAWS

Related links

References:

How to build a Big Data Pipeline

Azure Cloud

Onboarding : Azure Serverless

Topics

  • Key concepts
  • Azure Function App
  • Azure LogicApp

Key concepts

  • serverless: is Platform as a Service (PaaS)
  • functionapp
  • logicapp

Azure Function App

  • function app runs based on triggers
  • function app can be triggered by
    • webhook
    • API
    • Timer
    • Data Processing
  • can have more triggers
  • it’s event-driven
  • project files are
    • host.json
    • local.settings.json
  • runs code on-demand without explicitly provision / manage the infrastructure
  • hosting plans are
    • consumption plan : azure provisions all the necessary resources for running function and we pay as function is running
    • app service plan : just like web app. we can use the same plan with no additional costs
  • runtime stack
    • node js
    • .net core
    • java
    • powershell
  • <function-app-name>.azurewebsites.net
  • for stateful functions
  • function needs trigger, integration, price plan

Integration

  • Azure Cosmos DB
  • Event Hub
  • Event Grid
  • Notification Hub
  • Service Bus (Queue & Topic)
  • Storage (Blob, Table)
  • On-prem (using serrvice bus)
  • Twilio (SMS Message)

Event & triggers

  • Http
  • timer
  • cosmosdb
  • blob
  • queue
  • event grid
  • event hub (iot)
  • service bus queue
  • service bus topic
Consumption Plan (Pay for what used)App Service Plan (predictable monthly cost)Premium Plan
–Basic or higher tierimproved performance
Scaling is integrated in serviceScaling must be configuredVnet support
Pay for number of execution
Pay for CPU time & RAM
timeout after 5 min, iincreasable to 10 min
400,000 GB Free

Azure LogicApp

  • can have only one trigger
  • it’s event-driven

SQL

You owe your dreams your courage.

Koleka Putuma

Hobby-Future-Transportation

Autonomous Vehicles

Are people ready to adopt connected autonomous vehicles?

Shared driverless vehicles

The status of ADAS in the race to automation

MUSICC: open source repository for regulatory scenarios

Improving accessible transportation around the world

@CITY, automated cars and intelligent traffic in the city

Vehicle automation vs. vehicle autonomy – driving forces and technological impact

Autonomous shuttles and BRT: What purpose? What future?

Will autonomous driving improve road safety? certainty or wishful thinking?

Assurance at Five AI

Mobility as a Service: the training platform for autonomous fleets