It guarantees the types and shapes of data at runtime, making your code more robust and self-documenting.

Use cases:

• API request/response validation (e.g., FastAPI)

• Configuration management

• User input parsing

• Data cleaning from external sources (JSON, YAML, etc.)

pip install pydantic

Basic Model Usage

Create a BaseModel subclass

from pydantic import BaseModel

class User(BaseModel):
    id: int
    name: str
    email: str
    is_active: bool = True  # default value

Create an instance

# Create instance 
user = User(id=1, name="Alice", email="alice@example.com")
print(user)

Pydantic will automatically:

• Convert types ("1" to 1)

• Validate required fields

• Use default values where applicable

Example

user = User(id="42", name="Bob", email="bob@example.com")
print(user.id)  # 42 — automatically cast from string to int

Field Types and Constraints

You can add field constraints with Field():

from pydantic import Field

class Product(BaseModel):
    name: str
    price: float = Field(..., gt=0)
    description: str = Field(default="N/A", max_length=100)

product = Product(name="Laptop", price=-999.99)

Pydantic will raise an error

validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for Product
price
  Input should be greater than 0 [type=greater_than, input_value=-999.99, input_type=float]
    For further information visit https://errors.pydantic.dev/2.11/v/greater_than

Common Constraints

• gt, ge - Greater than, Greater or equal

• lt, le - Less than, Less or equal

• max_length - For strings

• min_items - For lists

• regex - Match a pattern

• default_factory - Create dynamic defaults

Nested Models

class Address(BaseModel):
    city: str
    country: str

class Employee(BaseModel):
    name: str
    address: Address

emp = Employee(name="Jane", address={"city": "Berlin", "country": "Germany"})
print(emp.address.city)  # "Berlin"

Aliases and Field Mapping

Sometimes JSON keys don't match your Python naming:

class Book(BaseModel):
    title: str = Field(..., alias='book_title')

book = Book(book_title="Deep Work")
print(book.title)

Custom Validators

from pydantic import validator

class CustomUser(BaseModel):
    name: str
    email: str

    @validator("email")
    def validate_email(cls, v):
        if "@" not in v:
            raise ValueError("Invalid email")
        return v

custom_user = CustomUser(name="Alice", email="alice#example.com")

Pydantic will raise error

validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for CustomUser
email
  Value error, Invalid email [type=value_error, input_value='alice#example.com', input_type=str]

Parsing & Serialization

Parsing from JSON/dict

data = {"id": "100", "name": "Alice", "email": "alice@example.com"}
user = User.model_validate(data)
print(user)

Serialization to JSON

print(user.model_dump())         # as dict
print(user.model_dump_json())    # as JSON string

This tutorial will guide you through:

1. What Type Hinting is and Why it's Important.

2. How to use mypy for static type checking.

3. How to use pyright for static type checking.

👉 What is Type Hinting and Why Use It?

Type Hinting involves adding annotations to your code to indicate the expected types of variables, function arguments, and return values. These hints are not enforced by the Python interpreter at runtime (Python remains dynamically typed), but they are crucial for static analysis tools.

Why is it important?

• Early Error Detection: Type checkers (like mypy and pyright) can analyze your code before it runs, catching potential type-related bugs that would otherwise only appear at runtime.

• Improved Readability and Maintainability: Type hints act as documentation, making it clearer what type of data a function expects and returns. This greatly helps other developers (and your future self!) understand and work with the code.

• Better IDE Support: Modern Integrated Development Environments (IDEs) and code editors leverage type hints for features like autocompletion, refactoring, and inline error checking.

• Refactoring Confidence: When refactoring code, type checkers can help ensure that changes don't introduce unintended type mismatches.

Basic Type Hinting Syntax:

Python's typing module provides many tools for type hinting.

from typing import List, Tuple, Dict, Optional, Union, Any

# 1. Basic types
def greet(name: str) -> str:
    return f"Hello, {name}!"

# 2. List of a specific type
def sum_numbers(numbers: List[int]) -> int:
    return sum(numbers)

# 3. Tuple with specific types at each position
def get_coordinates() -> Tuple[float, float]:
    return (10.5, 20.3)

# 4. Dictionary with specific key and value types
def get_user_info(user_id: int) -> Dict[str, str]:
    users = {1: {'name': 'Alice', 'email': 'alice@example.com'}}
    return users.get(user_id, {})

# 5. Optional type (can be a type or None)
def find_item(items: List[str], item: str) -> Optional[str]:
    if item in items:
        return item
    return None

# 6. Union type (can be one of several types)
def process_input(value: Union[str, int]) -> str:
    if isinstance(value, int):
        return str(value * 2)
    return value.upper()

# 7. Any type (use sparingly, means "any type")
def log_data(data: Any) -> None:
    print(f"Logging: {data}")

# Example usage (no type checking here, just function calls)
print(greet("World"))
print(sum_numbers([1, 2, 3]))
print(get_coordinates())
print(get_user_info(1))
print(find_item(['apple', 'banana'], 'banana'))
print(process_input("hello"))
print(process_input(123))
log_data({'key': 'value'})

👉 Static Type Checking with mypy

mypy is a popular static type checker for Python. It reads your code, analyzes the type hints, and reports any inconsistencies or potential type errors.

Installation:

pip install mypy

Basic Usage:

Let's create a file my_app.py with some code, including a deliberate type error:

# my_app.py
from typing import List

def calculate_average(numbers: List[int]) -> float:
    """Calculates the average of a list of integers."""
    if not numbers:
        return 0.0
    return sum(numbers) / len(numbers)

def process_data(data_list: List[str]) -> None:
    """Processes a list of strings."""
    for item in data_list:
        print(f"Processing: {item.upper()}")

# Correct usage
avg = calculate_average([10, 20, 30])
print(f"Average: {avg}")

process_data(["hello", "world"])

# Incorrect usage (type error)
# mypy should catch this!
process_data([1, 2, 3])

Now, run mypy from your terminal in the same directory:

mypy my_app.py

Expected mypy Output:

my_app.py:23: error: List item 0 has incompatible type "int"; expected "str"  [list-item]
my_app.py:23: error: List item 1 has incompatible type "int"; expected "str"  [list-item]
my_app.py:23: error: List item 2 has incompatible type "int"; expected "str"  [list-item]
Found 3 errors in 1 file (checked 1 source file)

Common mypy Flags:

• --strict: Enables all strict optional checks and warns about untyped definitions. Highly recommended for new projects.

• --ignore-missing-imports: Ignores imports that `mypy` cannot find type information for. Useful when libraries don't have type stubs.

• --disallow-untyped-defs: Reports an error whenever a function with no type annotations is encountered.

• --warn-unused-ignores: Warns if a `# type: ignore` comment is not necessary.

Configuration (mypy.ini or pyproject.toml):

# mypy.ini
[mypy]
python_version = 3.9
warn_unused_configs = True
warn_redundant_casts = True
warn_unused_ignores = True
disallow_untyped_defs = True
disallow_incomplete_defs = True
no_implicit_optional = True
strict_optional = True
check_untyped_defs = True
show_error_codes = True
pretty = True

# Exclude specific files or directories
exclude = venv/|docs/|build/

# Per-module options (e.g., for a specific library you use)
[mypy-my_app.*]
ignore_missing_imports = True

Then, just run mypy without specifying the file: “mypy .” (it will find the config).

👉 Static Type Checking with pyright

pyright is another powerful static type checker developed by Microsoft, primarily used with VS Code's Pylance extension, but also available as a standalone CLI tool. It's known for its speed and comprehensive type analysis.

Installation:

pip install pyright # As a Python package

Basic Usage:

Using the same my_app.py file from before:

pyright my_app.py

Expected pyright Output:

error: Argument of type "list[int]" cannot be assigned to parameter "data_list" of type "List[str]" in function "process_data"
    "Literal[1]" is not assignable to "str"
    "Literal[2]" is not assignable to "str"
    "Literal[3]" is not assignable to "str" (reportArgumentType)
1 error, 0 warnings, 0 informations

Configuration (pyrightconfig.json):

pyright uses a pyrightconfig.json file for configuration. Create this file in your project root:

{
    "include": ["."], // Check all files in the current directory
    "exclude": ["**/__pycache__", "venv", "docs", "build"],
    "reportMissingImports": true,
    "reportMissingTypeStubs": false, // Set to true if you want to be strict about stubs
    "reportUnusedImport": true,
    "reportUnusedVariable": true,
    "pythonVersion": "3.9",
    "typeCheckingMode": "strict" // "off", "basic", "strict"
}

Then, just run pyright without specifying the file: pyright (it will find the config).

👉 collections.namedtuple

namedtuple is part of Python's collections module. It's primarily used to create tuple-like objects that have named fields for each position, making the code more readable and self-documenting than plain tuples.

Key Characteristics:

• Immutable: Once a namedtuple instance is created, its values cannot be changed.

• Tuple Subclass: namedtuple objects are subclasses of tuple. This means they inherit all tuple behaviors (e.g., immutability, support for unpacking, iteration).

• Lightweight: Very low memory overhead, similar to regular tuples.

• Runtime Type Checking (Limited): Type hints are for static analysis only; Python doesn't enforce them at runtime.

• Access by Name or Index: You can access elements using dot notation (e.g., point.x) or by index (e.g., point[0]).

# tuple
t = (1, 2, 3)
print(t[0], t[1])

# namedtuple
from collections import namedtuple

# Define a namedtuple
Point = namedtuple('Point', ['x', 'y'])

# Create an instance
p = Point(10, 20)

print(p.x, p.y)  # Access by name
print(p[0], p[1]) # Access by index

⚡Immutability Sample

If you try to update a namedtuple member, it would result in an AttributeError

p.x = 30 # This would raise an AttributeError (immutable)

👉 typing.TypedDict

TypedDict is from the typing module and is used to define a dictionary type with a specific set of keys and their corresponding value types. It's primarily for static type checking (e.g., by tools like MyPy) and doesn't enforce types at runtime by default.

Key Characteristics:

• Mutable: TypedDict instances are regular dictionaries, so their values can be changed after creation. You can also add or remove keys.

• Dictionary Subclass (Conceptually): It behaves like a dictionary.

• Static Type Checking: Its main purpose is to provide type hints for dictionary structures, allowing type checkers to validate usage.

• Runtime Behavior: At runtime, a TypedDict behaves exactly like a normal dict.

• Required vs. Optional Keys: You can specify if keys are required or optional.

When to use TypedDict:

• When you need a dictionary-like structure with specific keys and types.

• When the primary goal is to improve code readability and enable static type checking for dictionary schemas.

• When you need mutability and the flexibility of a dictionary.

• For defining the expected structure of JSON data or API responses.

from typing import TypedDict

# Define a TypedDict
class User(TypedDict):
    name: str
    age: int
    email: str | None # Optional key

# Create an instance
user1: User = {'name': 'Alice', 'age': 30}
user2: User = {'name': 'Bob', 'age': 25, 'email': 'bob@example.com'}

print(user1)
print(user2)

user1['age'] = 31 # Mutable
print(user1)

⚡Static Type Checking

# user3: User = {'name': 'Charlie'} # MyPy would flag this as missing 'age'

👉 dataclasses

dataclasses (introduced in Python 3.7) provide a decorator that automatically generates special methods (like __init__, __repr__, __eq__) for classes primarily used to store data. They offer a good balance between the lightweight nature of namedtuple and the full flexibility of custom classes.

Key Characteristics:

• Mutable by Default: Data class instances are mutable by default, but you can make them immutable using frozen=True.

• Regular Class: They are regular Python classes, allowing you to add custom methods and properties.

• Type Hinting: Designed to work seamlessly with type hints, which are used by the decorator to generate methods.

• Readability: Reduces boilerplate code for data-holding classes.

• Runtime Type Checking (Not by default): Type hints are for static analysis, but you can add runtime checks if needed (e.g., using pydantic).

When to use dataclasses:

• When you need a class primarily for storing data.

• When you want to reduce boilerplate code for __init__, __repr__, __eq__, etc.

• When you need control over mutability (can be mutable or immutable).

• When you might want to add custom methods or properties later.

• For most general-purpose data structures in modern Python.

from dataclasses import dataclass

# Define a dataclass
@dataclass
class Product:
    name: str
    price: float
    quantity: int = 0 # Default value

# Create instances
p1 = Product("Laptop", 1200.0)
p2 = Product("Mouse", 25.50, 100)

print(p1)
print(p2)

p1.quantity = 5 # Mutable by default
print(p1)

# Make it immutable
@dataclass(frozen=True)
class ImmutablePoint:
    x: int
    y: int

ip = ImmutablePoint(5, 10)
# ip.x = 15 # This would raise a dataclasses.FrozenInstanceError

💥When to choose which

• namedtuple: For very simple, immutable data records where extreme memory efficiency is paramount, and you don't need custom methods or complex behavior.

• TypedDict: When you're primarily working with dictionary-like data (e.g., JSON payloads, API responses) and want to add type hints for better static analysis and code clarity, while retaining dictionary flexibility.

• dataclasses: For most modern Python applications where you need a clean, concise way to define data-holding classes. They offer a good balance of features, readability, and control over mutability, making them a versatile choice for structured data. If you find yourself writing __init__ and __repr__ methods just to store data, dataclasses are likely what you need.

We will learn via code and less by words.

A simple Pizza class

class Pizza:
    # constructor
    def __init__(self, toppings, size):
        # data members
        self.toppings = toppings
        self.size = size
    
    # methods
    def bake(self):
        print(f"Baking a {self.size}-inch pizza with {', '.join(self.toppings)} toppings!")

# Creating a pizza object (using the recipe)
my_pizza = Pizza(["cheese", "tomato", "pepperoni"], 12)
# calling the bake method
my_pizza.bake()

Example of Inheritance

Classic object-oriented hierarchy — readable, structured, and scalable.

class VeganPizza(Pizza):  # Inherits from Pizza
    # constructor
    def __init__(self, size):
        # super() is used to call the constructor of the parent class
        super().__init__(["tomato", "vegan cheese", "bell peppers"], size)
    
    # overriding the bake method
    def bake(self):
        print(f"Baking a {self.size}-inch VEGAN pizza with {', '.join(self.toppings)} toppings!")

# Creating a VeganPizza object
vegan_order = VeganPizza(10)
vegan_order.bake()

Polymorphism & Duck Typing

🦆 Duck Typing — "If it quacks like a duck, it is a duck"

bake_pizza expect the argument to have a bake method. That’s it!

No need for inheritance or interfaces.
If an object has a bake() method, we just trust it can bake.
"Don't check types. Just call methods."

# Inheirts from Pizza
class DeepDishPizza(Pizza):
    # overriding the bake method
    def bake(self):
        print(f"Baking a {self.size}-inch DEEP DISH pizza for 40 minutes!")

# Inheirts from Pizza
class ThinCrustPizza(Pizza):
    # overriding the bake method
    def bake(self):
        print(f"Baking a {self.size}-inch THIN CRUST pizza for 15 minutes!")

deep_dish = DeepDishPizza(["mozzarella", "tomato"], 14)
thin_crust = ThinCrustPizza(["cheese", "olives"], 12)

# Duck typing
def bake_pizza(pizza):
    pizza.bake()
    
# polymorphism
bake_pizza(deep_dish)
bake_pizza(thin_crust)

🤔 Why does this matter?

🔹 Inheritance gives structure — useful when building frameworks.
🔹 Duck typing gives flexibility — great for rapid, expressive code.

📌 Use inheritance when you want a formal hierarchy.
📌 Use duck typing when behavior matters more than ancestry.

👉 Bonus: Abstract Classes

Abstract classes work as templates, ensuring all subclasses follow a standard cooking method.

Example: Defining a Pizza Blueprint

from abc import ABC, abstractmethod

class AbstractPizza(ABC):  # Defining abstract class
    @abstractmethod
    def bake(self):
        pass  # Every subclass must implement this method

class Margherita(AbstractPizza):
    def bake(self):
        print("Baking Margherita pizza for 25 minutes!")

class Pepperoni(AbstractPizza):
    def bake(self):
        print("Baking Pepperoni pizza for 30 minutes!")

margherita = Margherita()
margherita.bake()

# A pure function always returns the same output for the same 
# input and has no side effects
def multiply(x, y):
    return x * y

# A pure function with safe side effects. Returns the same output 
# for the same input and does not mutate the input (immutablity)
def pure_double(list):
    new_list = list.copy()
    # double each element
    for i in range(len(new_list)):
        new_list[i] *= 2
    return new_list
 

# First-Class functions - functions can be assigned to variables or
# passed as arguments
def greet(name):
    return f'Hello {name}'

# First-Class functions. Higher-order function - function that takes
# a function as an argument
def executor(f, name):
    # execute the function
    return f(name)

print('Higher Order Function:',executor(greet, 'John'))

# First-Class functions - Assign function to a variable
message = greet
# use the function
print('Assign function to a variable:',message('John'))

# lambda functions
double = lambda x: x * 2
# use the function
print('Lambda function',double(5))


# Map, Filter, Reduce
from functools import reduce
numbers = [1, 2, 3, 4, 5]

# Map: Create a new list by applying a function to each item in an iterable
doubled = list(map(lambda x: x * 2, numbers))
print('Map function:',doubled)

# Filter: Create a new list by keeping only the items that pass a test
evens = list(filter(lambda x: x % 2 == 0, numbers))
print('Filter function:',evens)

# Reduce: Apply a function to a sequence of elements and return a single value (Aggregate)
sum = reduce(lambda x, y: x + y, numbers)
print('Reduce function:',sum)

# Decorator - function that takes a function as an argument and 
# returns a new function with added functionality
def logit(func):
    def with_logging(*args, **kwargs):
        print(func.__name__ + " was called")
        return func(*args, **kwargs)

    return with_logging

# Use the decorator
@logit
def add(x, y):
    return x + y

# Use the decorator
@logit
def sub(x, y):
    return x - y

# Use the decorated functions
print('Decorator:',add(2, 3))
print('Decorator:',sub(2, 3))

Intentionally did not add recursion, as in Python, things could get slower with recursion. Explanation:

Recursion in Python, especially in a functional style, can be slower than iterative approaches due to several factors. Each recursive call adds a new frame to the call stack, leading to function call overhead and increased memory usage. Python also has a recursion depth limit to prevent stack overflow errors, which can restrict the use of deep recursion.

Techniques like memoization (caching results of previous calls) and tail call optimization (converting recursion to iteration) can improve performance, but Python doesn't implement tail call optimization by default. Therefore, for performance-critical applications, iterative solutions are often preferred over recursive ones in Python.

This tutorial will introduce you to the core concepts of Pandas, focusing on its two primary data structures: Series and DataFrame. We'll explore common operations with practical examples, including loading data, inspecting it, selecting subsets, filtering, and performing basic aggregations.

Why Pandas?

• Powerful Data Structures: Series for 1D data and DataFrame for 2D tabular data.

• Easy Data Loading: Read data from various formats like CSV, Excel, SQL databases, JSON, and more.

• Flexible Data Manipulation: Tools for cleaning, transforming, merging, and reshaping data.

• Robust Analysis: Functions for statistical analysis, aggregation, and time series operations.

• Integration: Works seamlessly with other Python libraries like Matplotlib, Seaborn, and Scikit-learn.

Installation

If you don't have Pandas installed, you can do so using pip:

pip install pandas numpy matplotlib seaborn

Importing Pandas

It's standard practice to import Pandas with the alias pd:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns # We'll use this for some built-in datasets

1. Pandas Series: The 1D Data Structure

A Series is a one-dimensional labeled array capable of holding any data type (integers, strings, floats, Python objects, etc.). It's similar to a column in a spreadsheet or a SQL table, or a NumPy array with an added index.

Creating a Series:

# From a list
s1 = pd.Series([10, 20, 30, 40, 50])
print("Series from a list:\n", s1)

# From a list with a custom index
s2 = pd.Series([100, 200, 300], index=['a', 'b', 'c'])
print("\nSeries with custom index:\n", s2)

# From a dictionary (keys become the index)
data = {'apple': 10, 'banana': 20, 'cherry': 30}
s3 = pd.Series(data)
print("\nSeries from a dictionary:\n", s3)

# Accessing elements
print("\nElement at index 0 (s1):", s1[0])
print("Element at custom index 'b' (s2):", s2['b'])
print("Element at custom index 'cherry' (s3):", s3['cherry'])

# Slicing a Series
print("\nSlicing s1 (first 3 elements):\n", s1[:3])

2. Pandas DataFrame: The 2D Tabular Data Structure

A DataFrame is a two-dimensional labeled data structure with columns of potentially different types. It's like a spreadsheet, a SQL table, or a dictionary of Series objects. It's the most commonly used Pandas object.

Creating a DataFrame:

# From a dictionary of lists
data = {
    'Name': ['Alice', 'Bob', 'Charlie', 'David'],
    'Age': [25, 30, 35, 40],
    'City': ['New York', 'London', 'Paris', 'Tokyo']
}
df1 = pd.DataFrame(data)
print("DataFrame from a dictionary of lists:\n", df1)

# From a list of dictionaries
data_list = [
    {'Name': 'Eve', 'Age': 22, 'City': 'Berlin'},
    {'Name': 'Frank', 'Age': 28, 'City': 'Rome'},
    {'Name': 'Grace', 'Age': 33, 'City': 'Madrid'}
]
df2 = pd.DataFrame(data_list)
print("\nDataFrame from a list of dictionaries:\n", df2)

3. Loading Data into a DataFrame

One of Pandas' most powerful features is its ability to load data from various file formats.

Loading from a CSV file:

For this example, let's use the tips dataset, which is a common dataset for demonstrating data analysis. Seaborn provides convenient functions to load such datasets directly.

# Load a built-in dataset from Seaborn
tips = sns.load_dataset('tips')

# Quickly peek into the data with head and tail
print("\nFirst 5 rows of the 'tips' dataset:\n", tips.head())
print("\nLast 3 rows of the 'tips' dataset:\n", tips.tail(3))

# Inspecting DataFrames

4. Inspecting DataFrames

# Get basic information about the DataFrame
print("\nDataFrame Info:\n")
tips.info()

# Get descriptive statistics for numerical columns
print("\nDescriptive Statistics:\n", tips.describe())

# Get the shape (rows, columns) of the DataFrame
print("\nShape of DataFrame (rows, columns):", tips.shape)

# Get column names
print("\nColumn names:", tips.columns)

# Get data types of each column
print("\nData types:\n", tips.dtypes)

# Get unique values in a column
print("\nUnique days:", tips['day'].unique())

# Get the count of unique values in a column
print("\nValue counts for 'smoker':\n", tips['smoker'].value_counts())

5. Selecting Data (Columns and Rows)

Selecting Columns:

# Select a single column (returns a Series)
total_bill = tips['total_bill']
print("\n'total_bill' column (Series):\n", total_bill.head())

# Select multiple columns (returns a DataFrame)
subset_df = tips[['total_bill', 'tip', 'day']]
print("\n'total_bill', 'tip', 'day' columns (DataFrame):\n", subset_df.head())

Selecting Rows using loc and iloc:

• .loc[]: Label-based indexing (selects by row/column labels).

• .iloc[]: Integer-location based indexing (selects by integer position).

# Select row by integer position (iloc)
first_row = tips.iloc[0]
print("\nFirst row (iloc[0]):\n", first_row)

# Select rows by a range of integer positions (iloc)
rows_0_to_2 = tips.iloc[0:3] # rows 0, 1, 2
print("\nRows 0 to 2 (iloc[0:3]):\n", rows_0_to_2)

# Select specific rows and columns by integer position (iloc)
specific_cells = tips.iloc[[0, 2], [1, 3]] # Rows 0 and 2, columns 1 (tip) and 3 (sex)
print("\nSpecific cells (iloc[[0, 2], [1, 3]]):\n", specific_cells)

# Select rows by label (loc) - requires a named index, which 'tips' doesn't have by default.
# Let's set an index for demonstration
tips_indexed = tips.set_index('time')
print("\nTips DataFrame with 'time' as index:\n", tips_indexed.head())

# Now select by label
lunch_data = tips_indexed.loc['Lunch']
print("\nData for 'Lunch' (loc['Lunch']):\n", lunch_data.head())

# Select specific rows and columns by label (loc)
lunch_total_bill_tip = tips_indexed.loc['Lunch', ['total_bill', 'tip']]
print("\nTotal bill and tip for 'Lunch' (loc['Lunch', ['total_bill', 'tip']]):\n", lunch_total_bill_tip.head())

6. Filtering Data

Filtering allows you to select rows based on one or more conditions.

# Filter for tips greater than $5
high_tip_payers = tips[tips['tip'] > 5]
print("\nCustomers who paid more than $5 tip:\n", high_tip_payers.head())

# Filter for female smokers
female_smokers = tips[(tips['sex'] == 'Female') & (tips['smoker'] == 'Yes')]
print("\nFemale smokers:\n", female_smokers.head())

# Filter for tips on Sunday or Saturday
weekend_tips = tips[tips['day'].isin(['Sun', 'Sat'])]
print("\nTips on Sunday or Saturday:\n", weekend_tips.head())

7. Adding and Modifying Columns

# Add a new column: 'tip_percentage'
tips['tip_percentage'] = (tips['tip'] / tips['total_bill']) * 100
print("\nDataFrame with 'tip_percentage' column:\n", tips.head())

# Modify an existing column (e.g., round 'tip_percentage')
tips['tip_percentage'] = tips['tip_percentage'].round(2)
print("\nDataFrame with rounded 'tip_percentage':\n", tips.head())

# Create a new categorical column based on a condition
tips['generous_tipper'] = np.where(tips['tip_percentage'] > 20, 'Yes', 'No')
print("\nDataFrame with 'generous_tipper' column:\n", tips.head())

8. Handling Missing Data

Pandas uses NaN (Not a Number) to represent missing values.

# Create a DataFrame with missing values for demonstration
df_missing = pd.DataFrame({
    'A': [1, 2, np.nan, 4],
    'B': [5, np.nan, 7, 8],
    'C': [9, 10, 11, np.nan]
})
print("\nDataFrame with missing values:\n", df_missing)

# Check for missing values
print("\nMissing values (isnull):\n", df_missing.isnull())
print("\nTotal missing values per column:\n", df_missing.isnull().sum())

# Drop rows with any missing values
df_dropped_rows = df_missing.dropna()
print("\nDataFrame after dropping rows with NaN:\n", df_dropped_rows)

# Fill missing values with a specific value (e.g., 0)
df_filled_zero = df_missing.fillna(0)
print("\nDataFrame after filling NaN with 0:\n", df_filled_zero)

# Fill missing values with the mean of the column
df_filled_mean = df_missing.fillna(df_missing.mean())
print("\nDataFrame after filling NaN with column mean:\n", df_filled_mean)

9. Grouping and Aggregating Data

# Group by 'day' and calculate the mean of numerical columns
daily_avg = tips.groupby('day').mean(numeric_only=True)
print("\nDaily Averages:\n", daily_avg)

# Group by 'day' and 'sex' and calculate the sum of 'total_bill'
gender_day_total_bill = tips.groupby(['day', 'sex'])['total_bill'].sum()
print("\nTotal bill by day and gender:\n", gender_day_total_bill)

# Group by 'smoker' and calculate multiple aggregations
smoker_stats = tips.groupby('smoker').agg(
    avg_tip=('tip', 'mean'),
    max_bill=('total_bill', 'max'),
    count=('total_bill', 'count')
)
print("\nSmoker Statistics:\n", smoker_stats)

10. Merging and Joining DataFrames

# Create two dummy DataFrames
df_customers = pd.DataFrame({
    'customer_id': [1, 2, 3, 4],
    'name': ['Alice', 'Bob', 'Charlie', 'David']
})

df_orders = pd.DataFrame({
    'order_id': [101, 102, 103, 104, 105],
    'customer_id': [1, 3, 2, 1, 5], # Customer 5 does not exist in df_customers
    'amount': [50, 75, 20, 100, 30]
})

print("\nCustomers DataFrame:\n", df_customers)
print("\nOrders DataFrame:\n", df_orders)

# Inner merge (only common customer_ids)
merged_inner = pd.merge(df_customers, df_orders, on='customer_id', how='inner')
print("\nInner Merge (customers with orders):\n", merged_inner)

# Left merge (all customers, even if no orders)
merged_left = pd.merge(df_customers, df_orders, on='customer_id', how='left')
print("\nLeft Merge (all customers, with NaNs for no orders):\n", merged_left)

# Right merge (all orders, even if customer_id not in customers)
merged_right = pd.merge(df_customers, df_orders, on='customer_id', how='right')
print("\nRight Merge (all orders, with NaNs for unknown customers):\n", merged_right)

# Outer merge (all records from both, filling NaNs)
merged_outer = pd.merge(df_customers, df_orders, on='customer_id', how='outer')
print("\nOuter Merge (all records from both):\n", merged_outer)

What is Seaborn?

Seaborn is a Python data visualization library built on top of Matplotlib. While Matplotlib provides a solid foundation for creating a wide range of plots, Seaborn offers a higher-level interface with more aesthetic defaults and specialized functions for statistical plots. In essence, Seaborn makes it easier to create beautiful and informative visualizations with less code.

Key Features of Seaborn

• Simplified Syntax: Seaborn provides a concise and intuitive syntax, reducing the amount of code needed to create complex plots.

• Statistical Focus: Seaborn is designed to handle statistical data, offering functions for visualizing distributions, relationships between variables, and categorical data.

• Integration with Pandas: Seaborn seamlessly integrates with Pandas DataFrames, making it easy to visualize data directly from structured datasets.

• Attractive Defaults: Seaborn comes with a set of aesthetically pleasing default styles, making your plots look polished and professional.

• Customization: While Seaborn provides sensible defaults, it also allows for extensive customization to fine-tune the appearance of your plots.

Getting Started with Seaborn

To use Seaborn, you first need to install it. If you have Python and pip installed, you can simply run the following command in your terminal or command prompt:

pip install seaborn

I would highly recommend to use virtual environment

Once installed, you can import Seaborn and friends in your Python script or Jupyter Notebook:

import seaborn as sns
import matplotlib.pyplot as plt

Visualizing Data with Seaborn

Let's explore some of the fundamental Seaborn functions with examples using the "Iris" dataset, a classic dataset for classification tasks.

1. Visualizing Distributions

• Histograms: Histograms provide a visual representation of the distribution of a single variable.

  # Load the Iris dataset
  iris = sns.load_dataset('iris')
  
  # Create a histogram of the 'sepal_length' column
  sns.histplot(iris['sepal_length'], kde=True)  
  plt.xlabel('Sepal Length (cm)')
  plt.ylabel('Frequency')
  plt.title('Distribution of Sepal Length')
  plt.show()

  

  This code creates a histogram showing the frequency of different sepal lengths in the Iris dataset. The kde=True argument adds a kernel density estimate curve, providing a smoother representation of the distribution.

• Kernel Density Plots (KDEs): KDE plots provide a smooth estimate of the probability density function of a continuous variable.

  # Create a KDE plot of the 'sepal_width' column
  sns.kdeplot(iris['sepal_width'])
  plt.xlabel('Sepal Width (cm)')
  plt.ylabel('Density')
  plt.title('KDE Plot of Sepal Width')
  plt.show()

  

• Box Plots: Box plots display the distribution of data based on quartiles, showing the median, interquartile range, and potential outliers.

  # Create a box plot of 'petal_length' by 'species'
  sns.boxplot(x='species', y='petal_length', data=iris)
  plt.xlabel('Species')
  plt.ylabel('Petal Length (cm)')
  plt.title('Box Plot of Petal Length by Species')
  plt.show()

  This code generates a box plot comparing the distribution of petal lengths across different Iris species.

• Violin Plots: Violin plots are similar to box plots but provide a richer representation of the data distribution, showing the probability density of the data at different values.

  # Create a violin plot of 'petal_width' by 'species'
  sns.violinplot(x='species', y='petal_width', data=iris, inner='quartile')
  plt.xlabel('Species')
  plt.ylabel('Petal Width (cm)')
  plt.title('Violin Plot of Petal Width by Species')
  plt.show()

2. Visualizing Relationships Between Variables

• Scatter Plots: Scatter plots are used to visualize the relationship between two quantitative variables.

  # Create a scatter plot of 'sepal_length' vs. 'sepal_width'
  sns.scatterplot(x='sepal_length', y='sepal_width', hue='species', data=iris)
  plt.xlabel('Sepal Length (cm)')
  plt.ylabel('Sepal Width (cm)')
  plt.title('Scatter Plot of Sepal Length vs. Sepal Width')
  plt.show()

  The hue argument adds a third dimension to the plot by coloring the points based on the 'species' column.

  

• Regression Plots: Regression plots visualize the linear relationship between two variables and fit a regression line to the data.

  # Create a regression plot of 'sepal_length' vs. 'sepal_width'
  sns.regplot(x='sepal_length', y='sepal_width', data=iris)
  plt.xlabel('Sepal Length (cm)')
  plt.ylabel('Sepal Width (cm)')
  plt.title('Regression Plot of Sepal Length vs. Sepal Width')
  plt.show()

  

• Pair Plots: Pair plots visualize the relationships between all pairs of variables in a dataset.

  # Create a pair plot of the Iris dataset
  sns.pairplot(iris, hue='species')
  plt.title('Pair Plot of Iris Dataset')
  plt.show()

  

  Pair plots are a great way to quickly explore the relationships between multiple variables in your data.

3. Visualizing Categorical Data

• Count Plots: Count plots display the frequency of occurrences of different categories in a categorical variable.

  # Create a count plot of the 'species' column
  sns.countplot(x='species', data=iris)
  plt.xlabel('Species')
  plt.ylabel('Count')
  plt.title('Count Plot of Species')
  plt.show()

• Bar Plots: Bar plots are used to visualize the relationship between a categorical variable and a quantitative variable.

  # Create a bar plot of average 'sepal_length' by 'species'
  sns.barplot(x='species', y='sepal_length', data=iris)
  plt.xlabel('Species')
  plt.ylabel('Average Sepal Length (cm)')
  plt.title('Bar Plot of Average Sepal Length by Species')
  plt.show()

Conclusion

Seaborn is a powerful and versatile data visualization library that simplifies the process of creating beautiful and informative statistical plots in Python. Whether you're exploring distributions, visualizing relationships between variables, or analyzing categorical data, Seaborn provides a wide range of functions and customization options to meet your needs. By mastering Seaborn, you can effectively communicate your data insights and gain a deeper understanding of your datasets.

Shot-Based Prompting

We have "zero-shot", "one-shot", and "few-shot" prompting. The prefixs "zero", "one", and "few" refer to the number of input-output examples provided in the prompt to guide the LLM. This approach is also known as In-Context Learning (ICL), where the LLM learns directly from the examples within the prompt itself, without requiring explicit fine-tuning.

Zero-Shot Prompting:

Concept: In zero-shot prompting, you give the LLM a direct instruction to perform a task without providing any examples of the desired output. The model relies solely on its pre-trained knowledge to understand the task and generate a response.

Analogy: Imagine asking someone to translate a sentence into another language without ever showing them an example of translation. They have to rely on their general knowledge of languages and translation principles.

Example: The LLM is expected to use its understanding of English and French to produce the translation: "Bonjour, comment allez-vous?"

Prompt: Translate the following English text to French: "Hello, how are you?"

Use Cases: Suitable for simple and well-understood tasks that the LLM has likely encountered during its training, such as basic classification, general queries, and simple translations.

Advantages:

• Requires no task-specific examples, saving time and effort in prompt creation.

• Demonstrates the model's inherent capabilities and generalization from pre-training.

Limitations:

• Performance can be unpredictable or inaccurate for complex or nuanced tasks.

• The model might struggle with specific output formats or styles without guidance.

One-Shot Prompting:

Concept: One-shot prompting enhances zero-shot prompting by providing the LLM with a single example of the desired input-output format before the actual task instruction. This single example helps clarify the task and improve the model's understanding of what is expected.

Analogy: Now, before asking someone to translate, you show them one example of an English sentence and its French translation.

Example: The LLM sees the example "Good morning" -> "Bonjour" and then is asked to translate "How are you?". This single example can guide the model to provide a more accurate and appropriately formatted response.

Prompt: Translate English to French: English: Good morning French: Bonjour English: How are you? French:

Use Cases: Helpful for tasks that need more specific guidance or when the model struggles with ambiguity in a zero-shot setting, such as basic classification with specific labels or structured information extraction.

Advantages:

• Provides some context and clarifies the expected output format.

• Can lead to better performance than zero-shot prompting for slightly more complex tasks.

Limitations:

• A single example might not be sufficient for highly complex or nuanced tasks.

• The choice of the single example can significantly impact the model's output.

Few-Shot Prompting:

Concept: Few-shot prompting takes the idea of providing examples further by including multiple (typically 2-5) input-output examples in the prompt before the final task instruction. These examples help the model recognize patterns, understand complex tasks, and adhere to specific output formats or styles.

Analogy: You now provide several examples of English sentences and their French translations before asking for a new translation.

Example: The LLM learns from the provided examples how to associate certain phrases with "Positive" or "Negative" sentiment and can then apply this pattern to the new review.

Prompt: Classify the sentiment of the following movie reviews as positive or negative.  
Review: "This movie was amazing and I loved every second." Sentiment: Positive Review: "The plot was confusing and the acting was terrible." Sentiment: Negative 
Review: "I enjoyed it, but it wasn't anything special." Sentiment: Positive Review: "This film is a complete waste of time." Sentiment:

Use Cases: Best suited for complex tasks requiring specific output formats, nuanced understanding, or when higher accuracy and consistency are needed, such as generating structured outputs, handling nuanced classifications, or following specific writing styles.

Advantages:

• Significantly improves performance on complex tasks compared to zero-shot and one-shot prompting.

• Allows the model to learn from patterns and adhere to specific output requirements.

• Reduces the need for extensive fine-tuning data.

Limitations:

• Prompts can become long, potentially exceeding the model's context window limits.

• The quality and diversity of the examples are crucial for performance. Biased or poorly chosen examples can lead to suboptimal results.

• Can still be less effective than fine-tuning on a large, task-specific dataset for highly specialized tasks.

The Fine-Tuning Challenge

Traditional fine-tuning involves updating all parameters of a pre-trained model for a specific task. For context, modern LLMs can have billions or even trillions of parameters:

• GPT-3: 175 billion parameters

• LLaMA 2: Up to 70 billion parameters

Full fine-tuning these massive models requires:

• Enormous computational resources

• Significant GPU memory

• Substantial energy consumption

• Storage for multiple full model copies

For most organizations, this approach is prohibitively expensive and environmentally unsustainable.

Enter PEFT: The Efficient Alternative

PEFT techniques address these limitations by modifying only a small subset of parameters while keeping most of the pre-trained model frozen. This approach offers several advantages:

• Resource efficiency: Requires significantly less computing power

• Storage efficiency: Smaller parameter footprint

• Catastrophic forgetting prevention: Preserves general knowledge

• Adaptability: Easier to deploy across different tasks

Key PEFT Techniques Explained

Let's dive into several popular PEFT methods with their mathematical foundations:

1. LoRA (Low-Rank Adaptation)

LoRA focuses on representing parameter updates through low-rank decomposition matrices.

Mathematical concept: Instead of updating a weight matrix W directly, LoRA introduces two smaller matrices A and B:

ΔW = A·B

Where:

• A ∈ ℝ^(d×r)

• B ∈ ℝ^(r×k)

• r << min(d,k)

The modified forward pass becomes: y = x·(W + ΔW) = x·W + x·A·B

Example: For a 1000×1000 weight matrix (1M parameters), with rank r=8:

• A would be 1000×8 (8K parameters)

• B would be 8×1000 (8K parameters)

• Total trainable parameters: 16K (just 1.6% of original)

LoRA works exceptionally well for attention mechanisms in transformer architectures, often approaching full fine-tuning performance with just 0.1-1% of the parameters.

2. Prompt Tuning

Prompt tuning adds trainable continuous embeddings (soft prompts) to the input while keeping the model frozen.

Mathematical concept: For input tokens X, we prepend or append trainable embeddings P:

X' = [P; X] or X' = [X; P]

Where:

• X ∈ ℝ^(n×d) (n tokens with embedding dimension d)

• P ∈ ℝ^(p×d) (p trainable token embeddings)

Example: For a model with embedding dimension 768, adding 20 trainable prompt tokens means:

• P contains 20×768 = 15,360 trainable parameters

• For a 7B parameter model, this represents just 0.0002% of parameters

3. Prefix Tuning

Prefix tuning extends prompt tuning by adding trainable parameters to each layer of the model.

Mathematical concept: For a Transformer with L layers, we add prefixes Pi to each layer's key and value projections:

K'i = [Pk,i; Ki] V'i = [Pv,i; Vi]

Where:

• Pk,i and Pv,i ∈ ℝ^(p×d) are trainable prefixes for layer i

• Ki and Vi are the original key and value projections

Example: For a 12-layer transformer with 768-dimensional embeddings and 10 prefix tokens:

• Total trainable parameters: 12 layers × 2 (keys and values) × 10 tokens × 768 dimensions = 184,320 parameters

• Still a tiny fraction of the full model

4. Adapter Modules

Adapters insert small trainable modules between layers of the frozen model.

Mathematical concept: For a layer with transformation f, we insert an adapter module g:

y = f(x) + g(f(x))

Where g typically follows a bottleneck architecture: g(x) = W2·σ(W1·x)

With:

• W1 ∈ ℝ^(d×b) (down-projection)

• W2 ∈ ℝ^(b×d) (up-projection)

• b << d (bottleneck dimension)

• σ is a non-linear activation function

Example: For a transformer with hidden dimension 1024 and bottleneck dimension 64:

• Parameters per adapter: 1024×64 + 64×1024 = 131,072

• Adding adapters after each attention and FFN layer in a 12-layer model: ~3M parameters (much less than the full model)

5. BitFit

BitFit focuses exclusively on training the bias terms while keeping all other parameters frozen.

Mathematical concept: For a transformation with weights W and biases b: y = W·x + b

BitFit only updates b, keeping W frozen.

Example: In a 7B parameter model, bias terms might account for only ~0.1% of parameters (7M), making BitFit extremely parameter-efficient.

Mathematical Intuition Behind PEFT

The effectiveness of PEFT techniques lies in the concept of low intrinsic dimensionality. Despite having billions of parameters, the actual functional changes needed to adapt a model to a specific task often lie in a much lower-dimensional subspace.

Consider the parameter space ℝ^N of an LLM with N parameters. The task-specific adaptations often lie in a subspace ℝ^M where M << N. PEFT methods effectively find this lower-dimensional subspace, allowing efficient adaptation.

This can be formalized through Singular Value Decomposition (SVD) of the parameter update matrix: ΔW = UΣV^T

Where many singular values in Σ are close to zero, indicating that ΔW has a low effective rank.

Practical Applications and Examples

Example 1: Medical Domain Adaptation

Scenario: Adapting a general LLM for medical question answering

PEFT approach: LoRA with r=16

• Freeze all 7B parameters of the base model

• Train only 3M LoRA parameters (0.04% of full model)

• Training time: 4 hours on a single GPU vs. 1 week for full fine-tuning

• Performance: 96% of full fine-tuning accuracy

Example 2: Legal Document Analysis

Scenario: Fine-tuning for legal contract analysis

PEFT approach: Prefix tuning with 50 prefix tokens

• Train only 1.5M parameters

• Maintain model's general knowledge while specializing in legal terminology

• Adaptable across multiple jurisdictions with separate small prefix sets

Example 3: Multilingual Adaptation

Scenario: Adapting an English-centric LLM for low-resource languages

PEFT approach: Combination of adapter layers and prompt tuning

• Language-specific adapters (2M parameters per language)

• Shared cross-lingual prompt tokens (20K parameters)

• Results: Achieves 92% of full fine-tuning performance with only 0.03% of parameters

Recent Innovations in PEFT

QLoRA (Quantized LoRA)

QLoRA combines parameter quantization with LoRA, enabling fine-tuning of even larger models on consumer hardware.

Mathematical concept: The base model is quantized to 4 or 8 bits, while LoRA updates remain in full precision:

y = x·Q(W) + x·A·B

Where Q(W) is the quantized version of the original weights.

Example: Fine-tuning a 70B parameter model on a single consumer GPU with 24GB memory

• Base model quantized to 4-bit precision

• LoRA rank r=16

• 20M trainable parameters (0.03% of full model)

ULoRA (Unified LoRA)

ULoRA enables efficient transfer between different tasks by introducing task-specific vectors that modulate LoRA matrices.

Mathematical concept: For each task t, the LoRA update becomes:

ΔW_t = diag(v_t)·A·B

Where v_t is a learnable task vector that scales the contribution of different LoRA components.

Conclusion

Parameter-Efficient Fine-Tuning represents a critical development in making LLMs more accessible and practical. These techniques democratize access to state-of-the-art AI by reducing computational requirements while maintaining impressive performance.

The mathematical elegance of PEFT methods reveals a fundamental insight: adaptation often lies in low-dimensional subspaces of the parameter space. By identifying and focusing on these subspaces, we can efficiently specialize massive models for specific applications.

As LLMs continue to grow in size and capability, PEFT techniques will become increasingly essential for practical deployment, enabling organizations of all sizes to leverage the power of advanced AI while managing computational resources responsibly.

What problem does TF-IDF solve?

Imagine you're building a search engine or a text classification model. You have a large collection of documents, each with its own set of keywords. Your goal is to:

1. Identify the most important words in each document.

2. Compare the importance of these words across different documents.

The challenge lies in deciding which words are truly significant and which ones are just noise. This is where TF-IDF comes in – it helps you quantify the relevance of each word in a document, considering both its frequency (how often it appears) and rarity (how uncommon it is across all documents).

Term Frequency (TF)

The first part of TF-IDF is Term Frequency (TF), which measures how frequently a word appears within a single document. It's calculated as the number of times a word occurs in a document, divided by the total number of words in that document.

Mathematically, TF can be represented as:

TF(word, doc) = (number of occurrences of word in doc) / (total number of words in doc)

For example, if you have a document with 100 words and the word "cat" appears 5 times, its TF would be 5/100 = 0.05.

Inverse Document Frequency (IDF)

The second part is Inverse Document Frequency (IDF), which measures how rare or common a word is across all documents in the corpus. It's calculated as the logarithm of the ratio of the total number of documents to the number of documents containing that word.

Mathematically, IDF can be represented as:

IDF(word) = log(total number of documents / (number of documents containing word))

Using our previous example, if there are 1000 documents in the corpus and only 50 contain the word "cat", its IDF would be log(1000/50) ≈ 1.995.

Combining TF and IDF: TF-IDF

To create a meaningful measure of importance for each word in a document, you combine the Term Frequency (TF) with the Inverse Document Frequency (IDF). This is done by multiplying the two values together:

TF-IDF(word, doc) = TF(word, doc) * IDF(word)

In our example, the TF-IDF score for the word "cat" would be 0.05 * 1.995 ≈ 0.0995.

Interpretation and Applications

A high TF-IDF score indicates that a word is both frequent in a document (high TF) and rare across all documents (high IDF). This suggests that the word is important for distinguishing this particular document from others.

TF-IDF has numerous applications, including:

• Document classification: Use TF-IDF to select the most informative features (words) for a given document.

• Search engines: Implement TF-IDF-based ranking algorithms to retrieve relevant documents in response to user queries.

• Information retrieval: Utilize TF-IDF to identify the most representative words for a query or topic.

• Text summarization: Use TF-IDF to select the most important sentences or phrases from a document.

Limitations and Variations

While TF-IDF is a powerful technique, it has some limitations:

• Stopwords: Common words like "the", "and", etc., which are not informative, can skew TF-IDF scores.

• Overlapping documents: If multiple documents contain the same rare word, IDF may underestimate its importance.

To address these issues, you can modify the TF-IDF formula or use variations like:

• Logarithmic normalization: Apply a logarithmic transformation to the TF values to reduce the impact of very frequent words.

• Smoothing: Add a small constant to the IDFs to prevent zero divisions and improve robustness.

In conclusion, TF-IDF is a versatile technique for measuring the importance of words in a document, taking into account both their frequency and rarity across all documents. Its applications are diverse, ranging from search engines and text classification to information retrieval and summarization.

It is simple to get started and provides rapid development support via a wide range of widgets (buttons, sliders, dropdowns, etc.). Being a Python library it plays nicely with popular Python data science libraries like Pandas, NumPy, Matplotlib, Seaborn, Plotly, and scikit-learn.

Setting up the environment

• Create a new project directory and switch to it on a command-line/terminal

• Create a virtual environment

  pip -m venv venv

• Activate the virtual environment

  .\venv\Scripts\activate 

Hello from Streamlit

• Install the streamlit dependency.

  pip install streamlit

• Create a new Python file (e.g., main.py) and add a title and introductory text

  import streamlit as st
  
  st.title("Hello from Streamlit!")
  st.write("A simple Streamlit app to explore a dataset.")

• To run the app, use the command below (assuming your filename is main.py)

  streamlit run main.py

• This will automatically open a new tab in your web browser displaying your app. You should see the title and the text you added.

Working with data

Let's load a sample dataset. We'll use a simple CSV file. For this example, let's create a dummy CSV named sample_data.csv with the following content:

Name,Age,City,Score
Alice,25,New York,85
Bob,30,London,92
Charlie,22,Paris,78
David,35,Tokyo,95
Eve,28,New York,88

Let's load and display this data in our app:

import streamlit as st
import pandas as pd

st.title("Hello from Streamlit!")
st.write("A simple Streamlit app to explore a dataset.")

# Load the data
try:
    df = pd.read_csv("sample_data.csv")
except FileNotFoundError:
    st.error("Make sure 'sample_data.csv' is in the same directory.")
    st.stop()

# Display the raw data
st.subheader("Raw Data")
st.dataframe(df) # Or st.table(df) for a static table

The app will show "Raw Data" section with the DataFrame displayed. st.dataframe() provides an interactive table with sorting and searching capabilities. st.table() displays a static table.

Adding interactivity with widgets

Let's add some widgets to allow users to interact with the data.

• Checkbox: To show/hide the raw data.

• Selectbox: To filter data based on a column.

• Slider: To filter data based on a numerical range.

import streamlit as st
import pandas as pd

st.title("Hello from Streamlit!")
st.write("A simple Streamlit app to explore a dataset.")

# Load the data
try:
    df = pd.read_csv("sample_data.csv")
except FileNotFoundError:
    st.error("Make sure 'sample_data.csv' is in the same directory.")
    st.stop()

# Show/hide raw data
if st.checkbox("Show raw data"):
    st.subheader("Raw Data")
    st.dataframe(df)

# Filter by city
cities = df['City'].unique()
selected_city = st.selectbox("Filter by City:", ["All"] + list(cities))
if selected_city != "All":
    df_filtered_city = df[df['City'] == selected_city]
    st.subheader(f"Data for {selected_city}")
    st.dataframe(df_filtered_city)

# Filter by age
min_age = int(df['Age'].min())
max_age = int(df['Age'].max())
age_range = st.slider("Filter by Age:", min_age, max_age, (min_age, max_age))
df_filtered_age = df[(df['Age'] >= age_range[0]) & (df['Age'] <= age_range[1])]
st.subheader(f"Data for Ages {age_range[0]} to {age_range[1]}")
st.dataframe(df_filtered_age)

Now, the app will have checkboxes, a dropdown to select cities, and a slider to filter by age, dynamically updating the displayed data.

Visualizing Data

Let's create some visualizations to gain insights from the data. We'll use Matplotlib and Seaborn for this.

import streamlit as st
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

st.title("Hello from Streamlit!")
st.write("A simple Streamlit app to explore a dataset.")

# Load the data
try:
    df = pd.read_csv("sample_data.csv")
except FileNotFoundError:
    st.error("Make sure 'sample_data.csv' is in the same directory.")
    st.stop()

# Show/hide raw data
if st.checkbox("Show raw data"):
    st.subheader("Raw Data")
    st.dataframe(df)

# Filter by city
cities = df['City'].unique()
selected_city = st.selectbox("Filter by City:", ["All"] + list(cities))
df_filtered = df[df['City'] == selected_city] if selected_city != "All" else df

# Filter by age
min_age = int(df_filtered['Age'].min())
max_age = int(df_filtered['Age'].max())
age_range = st.slider("Filter by Age:", min_age, max_age, (min_age, max_age))
df_filtered = df_filtered[(df_filtered['Age'] >= age_range[0]) & (df_filtered['Age'] <= age_range[1])]

st.subheader("Filtered Data")
st.dataframe(df_filtered)

# --- Visualizations ---
st.subheader("Data Visualizations")

# Score distribution
st.subheader("Score Distribution")
fig_score, ax_score = plt.subplots()
sns.histplot(df_filtered['Score'], kde=True, ax=ax_score)
st.pyplot(fig_score)

# Age vs. Score
st.subheader("Age vs. Score")
fig_scatter, ax_scatter = plt.subplots()
sns.scatterplot(x='Age', y='Score', data=df_filtered, hue='City', ax=ax_scatter)
st.pyplot(fig_scatter)

# City counts
st.subheader("Number of People per City")
city_counts = df_filtered['City'].value_counts()
st.bar_chart(city_counts)

Here, we've added two visualizations: a histogram showing the distribution of scores and a scatter plot showing the relationship between age and score, colored by city. We use st.pyplot() to display Matplotlib figures and st.bar_chart() for a simple bar chart directly within Streamlit.

Layout and Organization

Streamlit provides several ways to organize your app's layout for better readability and structure.

• st.sidebar: To create a sidebar for filters and controls.

• st.columns: To arrange elements in multiple columns.

• st.expander: To hide content that users can expand if needed.

Let's incorporate a sidebar for our filters:

import streamlit as st
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

st.title("Hello from Streamlit!")
st.write("A simple Streamlit app to explore a dataset.")

# Load the data
try:
    df = pd.read_csv("sample_data.csv")
except FileNotFoundError:
    st.error("Make sure 'sample_data.csv' is in the same directory.")
    st.stop()

# --- Sidebar for Filters ---
st.sidebar.header("Filter Data")

# Filter by city in the sidebar
cities = df['City'].unique()
selected_city = st.sidebar.selectbox("Filter by City:", ["All"] + list(cities))
df_filtered = df[df['City'] == selected_city] if selected_city != "All" else df

# Filter by age in the sidebar
min_age = int(df_filtered['Age'].min())
max_age = int(df_filtered['Age'].max())
age_range = st.sidebar.slider("Filter by Age:", min_age, max_age, (min_age, max_age))
df_filtered = df_filtered[(df_filtered['Age'] >= age_range[0]) & (df_filtered['Age'] <= age_range[1])]

# Show/hide raw data in the main area
if st.checkbox("Show raw data"):
    st.subheader("Raw Data")
    st.dataframe(df)

# --- Main Area ---
st.subheader("Filtered Data")
st.dataframe(df_filtered)

# --- Visualizations ---
st.subheader("Data Visualizations")

# Score distribution
st.subheader("Score Distribution")
fig_score, ax_score = plt.subplots()
sns.histplot(df_filtered['Score'], kde=True, ax=ax_score)
st.pyplot(fig_score)

# Age vs. Score
st.subheader("Age vs. Score")
fig_scatter, ax_scatter = plt.subplots()
sns.scatterplot(x='Age', y='Score', data=df_filtered, hue='City', ax=ax_scatter)
st.pyplot(fig_scatter)

# City counts
st.subheader("Number of People per City")
city_counts = df_filtered['City'].value_counts()
st.bar_chart(city_counts)

Large Language Models have taken the tech world by storm, offering unprecedented natural language processing capabilities. While cloud-based services like ChatGPT have garnered much attention, there's a growing movement toward local, privacy-focused AI assistants. By running an LLM on your hardware, you maintain full control over your data and can customize the model to your specific needs.

Let's dive into how you can set up this game-changing tool using Ollama, an open-source framework for running LLMs locally.

Section 1: Preparing Your Battlestation

Before we summon our AI assistant, let's make sure our computer is ready for the task.

1.1 System Requirements

• A decent CPU (4+ cores recommended)

• At least 8GB of RAM (16GB+ for larger models)

• 10GB+ free disk space

• Windows 10/11, macOS, or Linux

1.2 Setting Up the Environment

We'll use Ollama as our LLM framework. There are official installers for all major platforms. We will be using Windows for this post.

The very first step is to download Ollama.

Pro Tip: To keep your C: drive clutter-free, we'll set up environment variables to store models elsewhere.

# PowerShell command to set OLLAMA_MODELS environment variable
[System.Environment]::SetEnvironmentVariable('OLLAMA_MODELS', 'D:\OllamaModels', [System.EnvironmentVariableTarget]::User)

Section 2: Summoning Your AI Assistant

Now that our battle station is primed, let's bring our AI assistant to life!

2.1 Installing Ollama

1. Install WSL2 if you're on Windows

2. Download and install Ollama following the instructions at ollama.ai

3. Open a terminal and run:

ollama run llama2

This command downloads and runs the Llama 2 model, a powerful general-purpose LLM.

2.2 Choosing Your AI Companion

Ollama supports various models. Here are some popular choices:

• Llama 2: Great all-rounder

• CodeLlama: Specialized for programming tasks

• Mistral: Excellent performance-to-size ratio

To run a specific model:

ollama run codellama

Section 3: Interfacing with Your AI Assistant

Now that your AI is alive and kicking, let's give it a proper interface!

3.1 Command-Line Interface

The simplest way to interact is through the terminal:

ollama run llama2 "Explain quantum computing in simple terms"

3.2 Graphical User Interfaces

For a more user-friendly experience, consider these open-source UIs:

• OpenWebUI: A sleek, web-based interface

• LM Studio: A desktop application with model management

Section 4: Customizing Your AI Sidekick

One of the biggest advantages of running your own LLM is customization. Let's explore how to tailor your AI assistant to your needs.

4.1 Fine-tuning

Fine-tuning allows you to specialize the model for specific tasks or domains. While advanced, it can significantly improve performance for your use cases.

4.2 Prompt Engineering

Craft effective prompts to guide your AI's responses. For example:

You are an expert software engineer. Analyze the following code for potential bugs and suggest improvements:

[Your code here]

Conclusion: Your AI Adventure Begins

Congratulations! You've just set up your personal AI assistant powered by cutting-edge LLM technology. With this powerful tool at your fingertips, you're ready to tackle complex problems, generate creative ideas, and boost your productivity like never before.

Remember, the key to getting the most out of your AI sidekick lies in experimentation and learning. Try different models, craft various prompts, and discover how this technology can best serve your unique needs.

Now, go forth and conquer with your new AI companion by your side! Share your experiences and creative uses in the comments below.

Pre-requisites

• Install Expo Orbit - https://expo.dev/orbit

• Install Expo Go - https://expo.dev/go

• Install Android Studio - You may also want to set up SDK and emulator, etc.

• Ensure Node.js is installed

• Ensure that VS Code, the favorite lightweight powerhouse, is installed.

• Follow - https://docs.expo.dev/tutorial/create-your-first-app/

Orbit will be very helpful for quickly starting your development environment.

Note: Windows defender or some anti-virus softwares may flag it as a virus. You would need to provide expception (if you trust the source).

Create an Expo React Native App

npx create-expo-app@latest <App Name>

The project is ready, and below, we have the output.

✅ Your project is ready!

To run your project, navigate to the directory and run one of the following npm commands.

- cd ReactBlast
- npm run android
- npm run ios # you need to use macOS to build the iOS project - use the Expo app if you need to do iOS development without a Mac
- npm run web

If it does not work out of the box for you, open Expo Orbit (installed earlier) and create and launch the emulator.

It worked like a charm for me on the first try.

Re-Starting your development environment

The easiest way is to start Expo Orbit. On Windows, it will be under hidden icons.

The elephant in the room: The Android Emulator

I had Android SDK set up, and emulators had already been created on my system; thus, it worked well.

You can see the “Emulators” in the image above. I hope you have already installed Android Studio.

Setting up the Android Emulator:

• Open Android Studio: Launch Android Studio.

• Open AVD Manager: Go to Tools > AVD Manager.

• Create Virtual Device: Click on "Create Virtual Device".

• Select Device Definition: Choose a device definition (e.g., Pixel 4).

• Select System Image: Choose an Android OS version (e.g., the latest API level) and download the system image if necessary.

• Configure and Finish: Configure the emulator settings (e.g., name, orientation) and click "Finish".

Using Expo Go

You can also use Expo Go. Just browse to the URL.

You can use “Install with orbit” option.

Last but not least: Re-Running the app

I found out that using the below command is the most friendly way of running your app.

npx expo start

When to Use Observer Pattern

• When changes to one object require notifying and updating multiple dependent objects.

• In event-driven architectures or pub-sub systems.

• To implement features like notifications, UI updates, or reactive programming.

Basic Observer Pattern Implementation

Define the Observer Interface

package com.agiledcoders.observer;

public interface Observer<T> {
    void update(T data);
}

The update method will receive the updates from the Subject.

Define the Subject Interface

package com.agiledcoders.observer;

public interface Subject<T> {
    void register(Observer<T> observer);
    void unregister(Observer<T> observer);
    void notify(T data);
}

• The register method allows one to register an Observer.

• The unregister method allows one to unregister an Observer.

• The notify method notifies the Observer(s)

Implement Subject

package com.agiledcoders.observer;

import java.util.ArrayList;
import java.util.List;
import java.util.Objects;

public class SubjectImpl implements Subject<String> {
    private final List<Observer<String>> observers = new ArrayList<>();

    @Override
    public void register(Observer<String> observer) {
        if (Objects.isNull(observer)) {
            throw new IllegalArgumentException("Observer must be non-null");
        }
        this.observers.add(observer);
    }

    @Override
    public void unregister(Observer<String> observer) {
        this.observers.remove(observer);
    }

    @Override
    public void notify(String data) {
        this.observers.forEach(stringObserver -> stringObserver.update(data));
    }
}

Implement Observer

package com.agiledcoders.observer;

public class ObserverImpl implements Observer<String> {
    private static int id;

    public ObserverImpl() {
        id = id + 1;
    }

    @Override
    public void update(String data) {
        System.out.println("----------------------------------------");
        System.out.println("Observer Id: " + id + " received update.");
        System.out.println(data);
        System.out.println("----------------------------------------");
    }
}

Create a Runner to test the pattern

package com.agiledcoders.observer;

public class Runner {
    public static void main(String[] args) {
        Observer<String> first = new ObserverImpl();
        Observer<String> second = new ObserverImpl();

        Subject<String> subject = new SubjectImpl();

        subject.register(first);
        subject.register(second);

        subject.notify("First iteration");

        subject.unregister(first);
        subject.notify("Second iteration");
    }
}

Output

----------------------------------------
Observer Id: 2 received update.
First iteration
----------------------------------------
----------------------------------------
Observer Id: 2 received update.
First iteration
----------------------------------------
----------------------------------------
Observer Id: 2 received update.
Second iteration
----------------------------------------

JUnit 5 (JUnit Jupiter) (Recommended)

<dependencies>
    <!-- JUnit 5 Dependency -->
    <dependency>
        <groupId>org.junit.jupiter</groupId>
        <artifactId>junit-jupiter</artifactId>
        <version>5.10.0</version> 
        <scope>test</scope>
    </dependency>
</dependencies>

JUnit 4 (For backward compatibility)

<dependencies>
    <!-- JUnit 4 Dependency -->
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.13.2</version> 
        <scope>test</scope>
    </dependency>
</dependencies>

The project structure and POM look as follows:

Verify JUnit integration

JUnit tests should be placed in the src/test/java directory of your Maven project.

1. Navigate to src/test/java.

2. Create a new test class if it is not already present (e.g., AppTest.java).

Example for JUnit 5

package org.example;

import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.assertEquals;

public class AppTest {
    @Test
    void testMultiplication() {
        int result = 2 * 3;
        assertEquals(6, result, "Result of multiplying 2 & 3 should be 6");
    }
}

Execute the test

mvn test

If all goes well, you will see output similar to the following:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.example.AppTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.040 s -- in org.example.AppTest
[INFO] 
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  2.055 s
[INFO] Finished at: 2024-12-27T13:27:44+05:30
[INFO] ------------------------------------------------------------------------

Clock Bound Wait

Clock-Bound Wait

A Clock-Bound Wait involves waiting for a certain time period or until a specific clock condition is met before an operation proceeds. This approach relies on the assumption that all nodes in the distributed system can synchronize time accurately enough to make coordinated decisions based on local clocks.

The Claimed Solution

Challenges

• Clock Drift:

  • Distributed nodes often experience clock drift, where their clocks slightly deviate from each other. Even with technologies like NTP (Network Time Protocol), perfect synchronization is challenging. This makes any clock-dependent coordination unreliable, especially over time.

• Lack of Global Clock:

  • Distributed systems inherently do not share a single, reliable global clock. Each node operates with its own clock, leading to time uncertainty across the system. Inconsistent clocks can cause actions to occur out of order or fail when they should succeed.

• Latency and Network Delays:

  • Network latency can add unpredictable delays to message transmission between nodes. If one node sets a clock-bound wait and other nodes have lagging or faster clocks, it could result in decisions being made prematurely or too late, leading to inconsistencies or timeouts.

• Imprecise Failures:

  • A node might wait for a certain time to pass based on its local clock, but if it doesn't accurately reflect the real time across the network, this could result in faulty decisions—especially in consensus algorithms (like Paxos or Raft) or in transactional systems.

Practical

• Assuming we have achieved some mechanism to calculate a “Correct Waiting Time” does that mean we can use this mechanism for a write-heavy system? No, as it would result in high latency and low-throughput.

• What mechanism would provide us with the correct waiting time? Will not we have a very chatty system?

Belief

There seems to be a belief that Count-Bound Wait is good for consistency.

Reality

Contrary to the general belief, Count-Bound systems lean towards eventual consistency and availability.

A system that heavily relies on clock-bound waits while boasting about consistency is indeed delusional. Why? Because:

1. Clock-bound waits sacrifice durability: If the system proceeds after a timeout without ensuring that all replicas have acknowledged the write, it can lead to inconsistencies.

2. No guarantee of write completeness: Clock-bound waits can't ensure that the write is durably stored across all required nodes.

3. Delayed reconciliation: Systems that rely on clock-bound waits will need to perform some form of eventual reconciliation to catch up with missed writes, introducing complexity.

Conclusion

Short-circuiting is a more general term that encompasses techniques like timeouts, buffer limits, and similar mechanisms. These are techniques designed to optimize for high availability (HA) and responsiveness when full acknowledgment from all nodes isn't possible or takes too long. AP systems (Availability-Partition Tolerant systems) are designed around eventual consistency, systems with a replication factor of N + 1/2 use timeouts to achieve this, and calling this a "clock-bound wait" is more of a semantic choice.

In other words, a clock-bound wait isn’t a revolutionary concept in itself—it’s just another form of short-circuiting, where the system "decides to wait" for a period of time before deciding to act based on the responses it receives.

In distributed systems, Clock-Bound Wait is typically not recommended due to the lack of reliable global clocks, potential clock drift, and network latencies. Instead, Versioning techniques like vector clocks, Lamport timestamps, or monotonically increasing versions are much more robust. They handle data consistency, conflict resolution, and causal relationships in an asynchronous and scalable manner, without relying on precise synchronization of clocks across distributed nodes.

To create multiple instances of a service, we usually think about port usage of these instances. Generally, you must use different ports if you try to do the same on your local machine.

For example, try to start multiple instances of this simple Go server below:

package main

import (
	"fmt"
	"log"
	"net"
	"net/http"
)

func GetLocalIP() string {
	addrs, err := net.InterfaceAddrs()
	if err != nil {
		return ""
	}
	for _, address := range addrs {
		// check the address type and if it is not a loopback the display it
		if ipnet, ok := address.(*net.IPNet); ok && !ipnet.IP.IsLoopback() {
			if ipnet.IP.To4() != nil {
				return ipnet.IP.String()
			}
		}
	}
	return ""
}

func helloHandler(w http.ResponseWriter, r *http.Request) {
	fmt.Fprintf(w, "Hello, World from %s!\n", GetLocalIP())
}

func main() {
	http.HandleFunc("/", helloHandler)
	log.Println("Starting Hello World server...")
	log.Fatal(http.ListenAndServe(":8888", nil))
}

The server returns a “Hello …” message with local IP information.

Running first instance

Below we are starting our Go server.

Check the process information bind to port 8888

$port = 8888
Get-Process -Id (Get-NetTCPConnection -LocalPort $port).OwningProcess

Try starting a new instance

go run server.go

Notice that the second time the terminal is not blocked as the process was already running.

Verify the process information

The process information is the same.

Change the port and try again

Change the port to 9999 and rerun the server.

The terminal is blocked

Verify the process information for port 9999

The process ID has changed.

Verify that the process for port 8888 is the same

Enters Docker

Docker’s internal networking allows services with the same port to coexist.

Docker Compose provides an enhanced setup to simulate clusters and fiddle with the distributed environments. As an engineer, this is gold.

Service Isolation

• Each service runs inside its own Docker container.

• They all expose the same port (say 8888 or 9999)

• They are isolated from each other because each container has its internal network stack.

Internal Docker Network

• Docker Compose automatically creates a virtual network for all the services defined in the docker-compose.yml file.

• Inside this network, each service can be reached by its service name (which you can define in the docker-compose file).

• They can run on the same port without conflict because the network namespace of each container is separate.

Port Binding

• We don't need to bind each service to a port on the host machine.

• Any other service composed in the same stack can communicate with other services in the same stack using Docker's internal network. Keep reading for example.

DNS

Docker Compose provides an automatic DNS resolution mechanism for services within the same network.

Automatic Network Creation

• When you run docker-compose up, Docker creates an isolated network (with its internal DNS) where all the services can communicate with each other.

• So, even if multiple containers use the same port internally (like a port 8080), they can do so because they each have their IP addresses within the Docker network.

• Docker automatically resolves service names (for example, server1, server2, etc.) to their corresponding container IPs. Thus, each service name acts like a hostname.

Service Discovery via Names:

• In the docker-compose.yml file, the names you give to services (like server1, server2 etc.) are used as DNS names inside the Docker network.

• Thus, any service inside the composed container can use it like a URI/URL.

Internal DNS Handling

• Docker handles the DNS resolution automatically.

• When an internal service tries to reach http://server1:8888, Docker’s internal DNS system will resolve server1 to the correct container’s IP address within the virtual network, ensuring traffic gets routed correctly.

Dockerfile for the Go server

# Official Go image
FROM golang:1.22.5-alpine

# Set current working directory (inside the container)
WORKDIR /app

# Copy go.mod and go.sum files (if you have any)
COPY go.mod go.sum ./

# Download dependencies
RUN go mod download

# Copy files
COPY ./src/goose/server.go .

# Build the Go app
RUN go build -o server .

# Expose port 8888
EXPOSE 8888

# Command to run the executable
CMD ["./server"]

PORT Sharing

To practically experience PORT sharing we will create a Reverse Proxy LB and use Named Docker service to distribute the traffic.

Reverse Proxy Load Balancer in Go

Here, we implement a basic reverse proxy that distributes incoming traffic to a list of backend services using a round-robin strategy. The LB will listen at the port 9090.

package main

import (
	"log"
	"net/http"
	"net/http/httputil"
	"net/url"
	"sync/atomic"
)

var backends = []string{
	"http://server1:8888",
	"http://server2:8888",
	"http://server3:8888",
}

var counter uint64

func getNextBackend() string {
	next := atomic.AddUint64(&counter, 1)
	return backends[(next-1)%uint64(len(backends))]
}

func reverseProxyHandler(w http.ResponseWriter, r *http.Request) {
	backendURL := getNextBackend()
	url, err := url.Parse(backendURL)
	if err != nil {
		http.Error(w, "Bad Gateway", http.StatusBadGateway)
		return
	}

	proxy := httputil.NewSingleHostReverseProxy(url)
	proxy.ServeHTTP(w, r)
}

func main() {
	http.HandleFunc("/", reverseProxyHandler)
	log.Println("Load Balancer running on port 9090...")
	log.Fatal(http.ListenAndServe(":9090", nil))
}

Docker file for load balancer - lb.Dockerfile

# Official Go image
FROM golang:1.20-alpine

# Set current working directory (inside the container)
WORKDIR /app

# (Standard) Copy go.mod and go.sum files (if you have any)
COPY go.mod go.sum ./

# (Standard) Download dependencies
RUN go mod download

# Copy files
COPY ./src/lb/lb.go .

# Build the Go app
RUN go build -o lb .

# Expose port 9090
EXPOSE 9090

# Command to run the executable
CMD ["./lb"]

Docker Compose Setup

We'll create a docker-compose.yml file to orchestrate the load balancer and the two backend services.

version: '3.8'

services:
    lb:
        build:
            context: .
            dockerfile: lb.Dockerfile
        ports:
            - '9090:9090'
        depends_on:
            - server1
            - server2
            - server3

    server1:
        build:
            context: .
            dockerfile: server.Dockerfile
        container_name: server1
        expose:
            - '8888'

    server2:
        build:
            context: .
            dockerfile: server.Dockerfile
        container_name: server2
        expose:
            - '8888'

    server3:
        build:
            context: .
            dockerfile: server.Dockerfile
        container_name: server3
        expose:
            - '8888'

networks:
    default:

Directory Structure

Below is my directory structure. I have multiple sample projects, so everything is under a “src” folder. The “…/” represents other code pieces and “goose” is the module directory where the “server.go” file contains the code for our server.

.
├── src/
│   ├── ...
│   ├── goose/
│   │   └── server.go
│   └── lb/
│       └── lb.go
├── go.mod
├── go.sum
├── server.Dockerfile
├── lb.Dockerfile
└── docker-compose.yml

Build and Run the Services

docker-compose up --build

Open your browser or use curl to access the load balancer on localhost:9090.

curl http://localhost:9090

You should see responses alternating between the three backend services, like:

Hello, World from 172.30.0.4!
Hello, World from 172.30.0.2!
Hello, World from 172.30.0.3!

• Real-Time Analytics and MongoDB

• Real-Time Analytics and MongoDB - The data processing

The Questions

Is ScyllaDB required for mid-size system? This is because Kafka consumers with fanout consumer proxy pattern can consume data fast enough and thus would keep the cost cheap.

How large scale systems can support near-real-time analytics?

Exploration

📌 Why Scylla Might Not Be Necessary for Mid-Size Systems

For mid-size systems, introducing ScyllaDB might be overkill. Kafka itself, with its fanout consumer proxy pattern, can be enough to handle a considerable amount of data processing. Here's why:

1. Kafka’s Strength in Mid-Scale Systems:

   • Kafka’s ability to fan out consumers allows you to scale horizontally with multiple processing applications reading from the same Kafka topic.

   • Each consumer can be responsible for specific transformations or pushing data into downstream databases, effectively handling real-time streaming at a mid-scale level without introducing an extra data storage layer like ScyllaDB.

2. Processing Directly from Kafka:

   • With Kafka Streams or Flink, you can perform most of the data processing directly from Kafka. This eliminates the need for another storage layer if your data processing requirements are simple enough and you don’t have to deal with extreme write throughput.

   • Since Kafka retains data for a configured time period (via topic retention policies), it can even handle some level of short-term persistence, making Scylla unnecessary unless you need high-write throughput over long periods.

📌 For Large-Scale Systems: The Kafka → Scylla → Kafka Chain

For large-scale systems where near-real-time processing is required, you would likely need a CDC (Change Data Capture) mechanism to keep data continuously flowing between ScyllaDB and Kafka. Here's a more detailed view of how this could work:

1. Kafka Ingestion:
   Kafka continues to serve as the data ingestion layer, capturing data from various upstream sources. This data could be raw events, logs, or user actions.

2. Initial Processing and Writing to ScyllaDB:
   Once the data is ingested, it can be transformed or enriched via Kafka Streams or a similar real-time processing framework.

   • Processed or partially processed data is then written to ScyllaDB, which acts as the primary persistent storage for the raw data.

3. Need for Continuous Data Flow (Near-Real-Time):
   In large-scale systems, periodic batch processing might not be enough, and you may need a more continuous data flow.

   • For low-latency data processing, you would need a CDC mechanism on ScyllaDB.

   • ScyllaDB doesn’t emit events like Kafka, but with CDC, you can track changes and push them back into Kafka for further real-time processing.

4. Kafka → Scylla → Kafka Flow:
   The Kafka → Scylla → Kafka chain could work as follows:

   • Data is first ingested and minimally processed in Kafka.

   • The processed data is stored in ScyllaDB for persistence and scalability.

   • A CDC mechanism captures any updates, changes, or new data in ScyllaDB.

   • Those CDC events are sent back to Kafka for real-time processing.

   • Kafka Streams or Flink can pick up these events and perform near-real-time processing, such as joining with other streams, aggregating, or further transforming the data.

5. Final Storage in a Read-Optimized Layer:
   After the real-time processing, the final transformed data is written to the read layer (such as MongoDB, Elasticsearch, or ClickHouse) where it’s optimized for queries, analytics, and reporting.

📌 When Would You Need This Setup?

You would need this Kafka → Scylla → Kafka chain in cases where:

• High write throughput needs to be sustained, but you also require real-time or near-real-time analytics.

• You want to separate raw data ingestion (Kafka and ScyllaDB) from complex processing logic (Kafka Streams, Flink) and query-optimized storage (MongoDB, Elasticsearch, etc.).

• The latency between raw data ingestion and queryable data needs to be minimized, and batch processing isn’t sufficient.

📌 Summary:

• For mid-size systems, Kafka and its fanout consumers can be sufficient without ScyllaDB, as it can handle streaming and simple transformations efficiently.

• For large-scale systems, ScyllaDB is useful for handling write-heavy workloads, but if you need continuous processing, the CDC mechanism is essential to avoid periodic batch jobs.

• The Kafka → Scylla → Kafka chain can provide real-time data flow for low-latency processing, with Kafka capturing ingestion events, Scylla storing raw data, and Kafka being used again for CDC-based continuous data processing.

The Question

How and why processing layer is reading data from ScyllaDB? The Kafka stream can transform or process data, or Kafka directly can be consumed by the processing layer. Also, Scylladb does not emit events like Kafka for processors to consume. So how does all this work?

Exploration

Let’s break down how the processing layer works in this architecture and clarify the roles of Kafka and ScyllaDB in the system.

📌 Kafka’s Role in Data Transformation

Kafka Streams (or similar processing frameworks) can transform data as it flows through Kafka. Kafka Streams allow for real-time data transformation, aggregation, and even windowed computations before the data is written to downstream storage (like ScyllaDB). However, Kafka typically excels at:

1. Ingestion: Capturing and buffering real-time data from producers.

2. Stream Processing: Kafka Streams or other processing frameworks (like Apache Flink, and Spark Streaming) can transform the data before pushing it to the next layer.

3. Fan-Out: Kafka’s design allows multiple consumers (or processing layers) to subscribe to the same topic, so the data can be processed in parallel by different applications.

But—Kafka doesn’t store data long-term, and it isn’t optimized for querying or analytics. That’s where ScyllaDB and MongoDB come in.

📌 ScyllaDB’s Role in the Architecture

ScyllaDB isn’t designed for event-driven processing like Kafka, so it doesn’t emit events to processors. Instead, ScyllaDB is used for high-throughput data storage of raw, unprocessed data (or minimally processed data). Here’s how that works in the context of the architecture:

1. Data Written from Kafka:
   Kafka Streams (or similar) pre-processes the raw data to some extent and then writes the partially transformed data to ScyllaDB. ScyllaDB acts as a time-series store or persistent store for fast writes, enabling storage of huge amounts of incoming data.

2. Data Querying for Further Processing:
   The processing layer (e.g., Apache Spark, Flink, or a custom service) can read the raw data stored in ScyllaDB and perform additional processing, cleaning, enrichment, or aggregation. This step is usually needed if more complex business logic, transformations, or computations are required before sending data to MongoDB.

   🔹 Querying from ScyllaDB:
   The processing layer can query ScyllaDB directly based on primary keys, clustering keys, or time-series filters. Since ScyllaDB is a highly scalable NoSQL database, you can retrieve and process large datasets very efficiently.

   🔹 Batch or Near Real-Time Processing:
   Although ScyllaDB doesn’t emit events, the processing layer can poll or query it periodically in batch mode (e.g., every few seconds or minutes) or use change data capture (CDC) if supported to detect new data in real time.

3. Further Processing in the Layer:
   Once the raw data is fetched from ScyllaDB, the processing layer can:

   • Aggregate, join, or clean the data (if needed).

   • Perform complex business logic transformations.

   • Enrich the data by combining it with data from other sources.

4. Processed Data Sent to MongoDB:
   After processing, the results are then sent to MongoDB. MongoDB is the final storage layer where the fully processed, enriched, and query-optimized data is stored. This is where real-time querying and analytics happen.

📌 How Kafka, ScyllaDB, and Processing Layers Interact:

1. Kafka Stream Processing:
   Kafka streams can handle some level of processing—filtering, basic transformation, and enrichment—but only to a certain point. Kafka primarily focuses on data ingestion and ensuring data flows reliably.

2. ScyllaDB for Low-Latency Storage:
   ScyllaDB handles the raw data that needs to be stored quickly and at scale. Its purpose is to allow your architecture to ingest huge amounts of data without worrying about losing data or overwhelming MongoDB, which is more suited for querying and analytics.

3. Processing Layer:
   The processing layer acts as the bridge between the raw write-heavy storage (ScyllaDB) and the read-optimized storage (MongoDB). It reads data from ScyllaDB, processes it (for example, aggregating hourly/daily statistics, and calculating metrics), and then sends the processed data to MongoDB for querying.

   In this setup, the processing layer can be implemented using:

   • Apache Spark or Flink for distributed data processing.

   • Custom microservices that pull data from ScyllaDB, process it, and push to MongoDB.

📌 Benefits of This Architecture:

1. Separation of Concerns:
   Kafka handles ingestion and streaming, ScyllaDB handles write-heavy storage, and MongoDB handles read-heavy querying and analytics.

2. Optimized for Scale:
   ScyllaDB’s design allows it to scale linearly and handle large amounts of data with low-latency writes, while MongoDB is optimized for real-time analytics at scale.

3. Flexibility in Processing:
   The processing layer can handle complex transformations and enrichment before the data reaches MongoDB, making it highly adaptable to different business needs.

4. Efficient Resource Usage:
   By offloading high-throughput writes to ScyllaDB, you avoid the costs associated with MongoDB’s sharded clusters, which would be far more expensive for write-heavy workloads.

The Problem

An analytical system needs to ingest data at a high rate. This requires the storage system to have low write latency and high throughput.

MongoDB is not always the best choice for write-heavy systems compared to other NoSQL databases like Cassandra or ScyllaDB. Here are some challenges:

📌 Challenges of MongoDB in Write-Heavy Systems

1. Write Performance: MongoDB’s default write operations are not as optimized for high-throughput writes as some other NoSQL databases. Databases like Cassandra and ScyllaDB offer better performance for write-heavy workloads due to their design, which allows for low-latency writes and linear scalability across nodes.

2. Clustering Costs: Managing a sharded MongoDB cluster can become costly. Sharding requires not only data but also indexes to be duplicated across shards to ensure proper performance and consistency during queries. Each shard holds its own copy of the index, increasing storage and resource consumption.

3. Critical Shard Key Selection: Choosing the right shard key is essential in MongoDB. A poorly chosen shard key can lead to hot spots, uneven distribution of data, or poor performance as some nodes may be overloaded while others remain underutilized.

📌 Could MongoDB be a Good Fit for Read-Heavy Analytics?

Yes, using MongoDB for storing processed data can work well in a read-heavy real-time analytics system, but pairing it with a write-optimized system can balance the workload.

🔹 Kafka or Scylla for Write-Heavy Systems:
Kafka or Scylla could handle high-throughput writes and act as a buffer, ingesting data in real-time. Kafka would be useful for stream processing, and ScyllaDB could store low-latency, write-heavy data due to its high performance in write operations.

🔹 MongoDB for Processed Data Storage:
Once the data is processed (aggregated, cleaned, transformed), you can store it in MongoDB for real-time querying and analysis. MongoDB excels in complex queries and can efficiently handle real-time analytics workloads when the focus shifts from writing to reading large volumes of data.

The Proposal

👉 Let’s dive deeper into a hybrid architecture combining Kafka, ScyllaDB, and MongoDB for handling high-throughput, low-latency writes while optimizing for real-time analytics.

📌 Proposed Architecture Overview

The architecture focuses on decoupling the write-intensive and read-intensive components to optimize both data ingestion and real-time analytics.

1. Data Ingestion Layer with Kafka

At the heart of this architecture is Apache Kafka or a similar message queue system that can act as a real-time data ingestion layer. Kafka ensures that incoming data is reliably captured and distributed to downstream components.

🔹 Kafka’s Role: Kafka can buffer and stream real-time data from various sources (e.g., user activity, sensor data, logs) while allowing multiple consumers to process this data in parallel. Its ability to handle millions of writes per second makes it an ideal fit for high-throughput applications.

🔹 Advantages:

• Low Latency: Kafka provides real-time message delivery with low overhead.

• Fault Tolerance: Kafka is highly reliable and can ensure that no data is lost even when downstream systems are under load.

• Horizontal Scalability: Kafka brokers can be scaled horizontally to match increasing data volumes.

🔹 Example Use Case:
For a real-time IoT system, Kafka can collect data from thousands of sensors streaming in real-time, such as temperature, humidity, and device statuses.

2. Write-Heavy Data Store with ScyllaDB

Once Kafka ingests data, the raw, unprocessed data needs to be stored in a write-optimized NoSQL system. This is where ScyllaDB (or Cassandra) comes in. ScyllaDB is a powerful NoSQL database designed for low-latency, high-write performance, perfect for write-heavy workloads.

🔹 Why ScyllaDB (or Cassandra):

• Low-Latency Writes: ScyllaDB’s design supports high throughput with low-latency writes, perfect for rapidly incoming data from Kafka.

• Linear Scalability: ScyllaDB is known for its linear scaling, meaning that adding more nodes will proportionally improve the performance of writes and reads.

• Automatic Data Replication: ScyllaDB automatically handles replication and fault tolerance, ensuring your data is safe and consistent across multiple nodes.

🔹 Architecture:

• Kafka Consumers: A set of consumers reads messages from Kafka and writes them to ScyllaDB in real-time.

• Time-Series Data: ScyllaDB excels at time-series data, where each incoming data point (e.g., from IoT sensors) is written as a new record.

🔹 Example Use Case:

• In a real-time analytics system for a ride-hailing service, ScyllaDB can store raw trip data (e.g., location updates, timestamps) from millions of drivers and riders. It’s write-intensive, but it doesn’t require real-time querying just yet.

3. Real-Time Analytics with MongoDB

Once data is stored in ScyllaDB, it can be processed (aggregated, cleaned, transformed) and then passed to MongoDB, which acts as the read-optimized store for real-time analytics. This is where MongoDB’s strength lies—complex, flexible querying and serving real-time insights.

🔹 Why MongoDB:

• Schema Flexibility: MongoDB allows developers to store processed data in a flexible schema, making it ideal for evolving business needs.

• Rich Querying: MongoDB’s aggregation framework is powerful, enabling real-time insights by executing complex queries, filtering, and grouping operations.

• Scaling Reads: MongoDB can efficiently scale horizontally to handle heavy read loads, which is critical for real-time analytics.

• Indexes: MongoDB’s powerful indexing system supports fast querying. However, index duplication in sharded clusters can increase resource usage, so careful consideration of index design and shard keys is essential.

🔹 Sharding and Indexing:

• MongoDB uses sharding to distribute large datasets across multiple nodes. But each shard holds its own index, which duplicates storage and indexing overhead.

• The shard key is critical—choosing the right shard key affects both performance and distribution of data across nodes.

🔹 Example Use Case:

• For the same ride-hailing service, MongoDB can store processed trip data such as total distance, fare, driver ratings, and customer feedback. The aggregation of this data could provide real-time insights such as average trip length, busiest times of day, or heatmaps of ride demand.

📌 The Role of Each Component

1. Kafka (Data Ingestion):
Kafka ingests real-time streaming data (e.g., logs, sensor data, app activity) and acts as the intermediary between the write-heavy store and processing layer. Kafka scales horizontally to handle high throughput and ensures no data is lost under heavy load.

2. ScyllaDB (Write-Optimized Store):
ScyllaDB is the system of choice for handling low-latency, write-heavy workloads. It acts as a scalable time-series store for unprocessed data that needs to be written at high speed. While MongoDB would struggle with this volume of writes, ScyllaDB excels here.

3. MongoDB (Read-Optimized Analytics):
Once the data is processed, MongoDB provides a read-optimized storage layer that supports flexible schema changes and complex queries, ideal for real-time analytics. It scales horizontally to handle massive datasets for fast, real-time querying.

📌 Architecture Flow Example:

1. Data Ingestion: Real-time data from IoT devices, apps, or logs flows into Kafka.

2. Low-Latency Writes: Kafka streams the raw data into ScyllaDB, where it is stored and partitioned based on time, location, or other business-specific criteria.

3. Processing Layer: A data processing layer reads data from ScyllaDB, processes it (e.g., aggregates, cleans, transforms), and passes the results to MongoDB.

4. Real-Time Analytics: MongoDB stores the processed data and provides real-time insights via its rich querying and aggregation framework. Dashboards, analytics platforms, and apps pull data from MongoDB to display insights.

📌 Advantages of This Hybrid Setup:

🔹 Decoupling Write and Read Workloads:
By splitting write-heavy and read-heavy operations between ScyllaDB and MongoDB, the architecture ensures that each database handles the workload it’s optimized for.

🔹 Kafka’s Buffering Capabilities:
Kafka acts as a buffer, making the system more resilient to spikes in traffic, ensuring that the databases can ingest data at their own pace.

🔹 Scalability:
Both ScyllaDB and MongoDB are designed for horizontal scalability, making it easier to scale the system as data volumes grow.

🔹 Optimized Costs:
MongoDB clustering is costly, especially when handling high write volumes. By offloading writes to ScyllaDB, you reduce MongoDB’s storage and processing burden, which can significantly cut down on operational costs.

📌 Challenges to Consider:

1. Data Synchronization:
   You’ll need an efficient processing pipeline between ScyllaDB and MongoDB to ensure that data is processed and synced in near real-time.

2. Shard Key Design:
   Choosing the right shard key in MongoDB is critical. Poor shard key selection can lead to uneven data distribution or “hotspots,” which can degrade performance.

3. Operational Complexity:
   Managing multiple database systems increases the operational overhead. Ensuring seamless integration and monitoring between Kafka, ScyllaDB, and MongoDB will require careful planning and tooling.

📌 Conclusion

This hybrid architecture provides a balance between low-latency writes (with ScyllaDB) and real-time analytics (with MongoDB), powered by Kafka for reliable data ingestion. This approach maximizes performance while managing the cost and complexity associated with MongoDB clustering.

👉 For mid-size systems introducing ScyllaDB might be overkill. Kafka itself, with its fanout consumer proxy pattern, can be enough to handle a considerable amount of data processing.