Skip to main content

27 posts tagged with "python"

View All Tags

The Complete Python Code Quality Toolchain: Mypy, Flake8, Black, and Pylint

· 17 min read

Introduction: Why Code Quality Matters​

┌─────────────────────────────────────────────────────────────────────────────┐
│ THE PYTHON QUALITY ECOSYSTEM │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ BLACK │ │ FLAKE8 │ │ MYPY │ │ PYLINT │ │
│ │ The │ │ The │ │ The │ │ The │ │
│ │ Stylist │ │ Police │ │ Engineer │ │ Inspector │ │
│ │ (Auto) │ │ (Style) │ │ (Types) │ │ (Design) │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │ │ │ │ │
│ "Make it" "Is it" "Will it" "How good" │
│ pretty clean? break? is it?" │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ TOGETHER THEY ENSURE: │ │
│ │ • Professional Code • Fewer Bugs • Easier Maintenance │ │
│ │ • Team Consistency • Better Design • Faster Onboarding │ │
│ │ • Code Review Speed • Reliability • Long-term Sustainability │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ For Financial Code (Your Tax Project): │
│ • Type Safety = Correct Calculations │
│ • Clean Code = Understandable Tax Logic │
│ • Good Design = Maintainable Financial Rules │
│ • Consistency = Team Collaboration │
└─────────────────────────────────────────────────────────────────────────────┘

1. The Four Pillars Explained​

1.1 BLACK: The Opinionated Formatter​

Concept: Black is like an auto-formatting machine - it takes your code and makes it consistent without debate.

┌─────────────────────────────────────────────────────────────────────────────┐
│ BLACK: BEFORE AND AFTER │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ BEFORE BLACK: Inconsistent style │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ def calculateTax(items,tax_rate): │ │
│ │ total=0 │ │
│ │ for item in items: │ │
│ │ total=total+item │ │
│ │ return total*(1+tax_rate) │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ⬇ Black Formatting │
│ │
│ AFTER BLACK: Consistent, PEP 8 compliant │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ def calculate_tax(items, tax_rate): │ │
│ │ total = 0 │ │
│ │ for item in items: │ │
│ │ total = total + item │ │
│ │ return total * (1 + tax_rate) │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ Benefits: No more style debates! Saves time in code reviews. │
└─────────────────────────────────────────────────────────────────────────────┘

Key Features:​

  • No configuration needed (but configurable if you must)
  • Deterministic - same code always formats the same way
  • Fast - processes thousands of lines per second
  • PEP 8 compliant - follows Python style guide
  • Safe - only changes whitespace and formatting

Installation & Usage:​

# Install
pip install black

# Format a file
black my_file.py

# Format directory
black src/

# Check without formatting
black --check src/

# With line length (default 88)
black --line-length 100 src/

# Format Jupyter notebooks
black --line-length 88 notebook.ipynb

Configuration (pyproject.toml):​

[tool.black]
line-length = 88
target-version = ['py311']
include = '\.pyi?$'
extend-exclude = '''
/(
\.eggs
| \.git
| \.hg
| \.mypy_cache
| \.tox
| \.venv
| _build
| buck-out
| build
| dist
)/
'''

1.2 FLAKE8: The Style Enforcer​

Concept: Flake8 is like a traffic cop - it enforces the rules of the road (PEP 8) and catches obvious violations.

┌─────────────────────────────────────────────────────────────────────────────┐
│ FLAKE8: WHAT IT CHECKS │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │
│ │ PEP 8 │ │ PyFlakes │ │ McCabe │ │
│ │ Style Rules │ │ Logical Errors │ │ Complexity │ │
│ │ • Line length │ │ • Unused vars │ │ • Deep nesting │ │
│ │ • Spacing │ │ • Undefined │ │ • Many returns │ │
│ │ • Naming │ │ • Duplicate │ │ • Many branches│ │
│ └──────────────────┘ └──────────────────┘ └──────────────────┘ │
│ │
│ Example Violations: │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ import os, sys # E401: Multiple imports │ │
│ │ x=5+3 # E225: Missing spaces │ │
│ │ def bad():pass # E701: Multiple statements │ │
│ │ very_long_name = "This line is way too long..." # E501: Line length│ │
│ │ unused = 42 # F841: Unused variable │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘

Core Components:​

  1. pycodestyle - PEP 8 style checking
  2. pyflakes - Logical error detection
  3. mccabe - Code complexity analysis

Installation & Usage:​

# Install
pip install flake8

# With popular plugins
pip install flake8 flake8-docstrings flake8-bugbear flake8-comprehensions

# Basic usage
flake8 my_file.py

# Check directory
flake8 src/

# Show statistics
flake8 --statistics src/

# Output as JSON
flake8 --format=json src/ > report.json

Configuration (.flake8):​

[flake8]
max-line-length = 88
extend-ignore = E203, W503 # Conflicts with Black
exclude = .git,__pycache__,build,dist,.venv,venv
per-file-ignores =
__init__.py:F401 # Allow unused imports in __init__
tests/*:S101 # Allow assert in tests
select = E,F,W,B,C,D # E: errors, F: pyflakes, W: warnings, B: bugbear, C: complexity, D: docstrings
max-complexity = 10

1.3 MYPY: The Type Safety Engineer​

Concept: Mypy is like a building inspector - it checks the structural integrity (types) before anyone moves in.

┌─────────────────────────────────────────────────────────────────────────────┐
│ MYPY: STATIC TYPE CHECKING DEMO │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ WITHOUT MYPY: Runtime crash │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ def calculate_tax(income, rate): │ │
│ │ return income * rate │ │
│ │ │ │
│ │ # User passes string accidentally │ │
│ │ tax = calculate_tax("50000", 0.2) # Runtime: TypeError! │ │
│ │ │ │
│ │ # Error only discovered when code runs │ │
│ │ # Could be in production! │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ WITH MYPY: Error caught before runtime │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ def calculate_tax(income: float, rate: float) -> float: │ │
│ │ return income * rate │ │
│ │ │ │
│ │ # Mypy catches error immediately │ │
│ │ tax = calculate_tax("50000", 0.2) │ │
│ │ ↑ │ │
│ │ # Error: Argument 1 has incompatible type "str" │ │
│ │ # expected "float" │ │
│ │ │ │
│ │ # Fix it before it becomes a problem │ │
│ │ tax = calculate_tax(50000.0, 0.2) # ✅ Correct │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘

Type System Features:​

  • Type hints - Explicit type declarations
  • Type inference - Automatic type detection
  • Generics - Type parameters for containers
  • Union types - Multiple possible types
  • Optional types - Values that can be None
  • Protocols - Structural typing (duck typing)

Installation & Usage:​

# Install
pip install mypy

# Basic usage
mypy my_file.py

# Check with missing imports
mypy --ignore-missing-imports src/

# Strict mode (all checks)
mypy --strict src/

# Generate HTML report
mypy --html-report mypy_report src/

Configuration (mypy.ini):​

[mypy]
python_version = 3.11
warn_return_any = true
disallow_untyped_defs = true
disallow_incomplete_defs = true
check_untyped_defs = true
disallow_untyped_decorators = true
no_implicit_optional = true
warn_redundant_casts = true
warn_unused_ignores = true

[mypy-tests.*]
ignore_missing_imports = true

1.4 PYLINT: The Comprehensive Inspector​

Concept: Pylint is like a home inspector - it examines everything from foundation to roof and gives you a detailed report.

┌─────────────────────────────────────────────────────────────────────────────┐
│ PYLINT'S INSPECTION AREAS │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ SCORE: 8.5/10 │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ 🔵 CONVENTION (C) - Style & Standards │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ • C0103: bad-name "calculateTax" → "calculate_tax" │ │
│ │ • C0114: missing-module-docstring │ │
│ │ • C0301: line-too-long (120/88) │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ 🟡 REFACTOR (R) - Design & Smells │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ • R0913: too-many-arguments (8/5) │ │
│ │ • R0902: too-many-instance-attributes (10/7) │ │
│ │ • R1705: unnecessary-else-after-return │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ 🟠 WARNING (W) - Python Issues │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ • W0621: redefined-outer-name │ │
│ │ • W0703: broad-except │ │
│ │ • W0613: unused-argument │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ 🔴 ERROR (E) - Probable Bugs │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ • E0602: undefined-variable │ │
│ │ • E1121: too-many-function-args │ │
│ │ • E1136: unsubscriptable-object │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘

Key Analysis Areas:​

  1. Convention (C) - PEP 8 and naming standards
  2. Refactor (R) - Code smells and design issues
  3. Warning (W) - Python-specific problems
  4. Error (E) - Probable bugs
  5. Fatal (F) - Pylint internal errors

Installation & Usage:​

# Install
pip install pylint

# Basic usage
pylint my_file.py

# Directory analysis
pylint src/

# Output formats
pylint --output-format=json my_file.py
pylint --output-format=colorized my_file.py

# With reports
pylint --reports=y my_file.py

# Fail under score threshold
pylint --fail-under=8.0 src/

Configuration (.pylintrc):​

[MASTER]
python-version=3.11
paths=src/

[MESSAGES CONTROL]
disable=
missing-docstring,
too-few-public-methods,
broad-except

[DESIGN]
max-args=5
max-attributes=7
max-branches=12
max-methods=20

[FORMAT]
max-line-length=88

2. Comparison: The Complete Toolchain​

┌─────────────────────────────────────────────────────────────────────────────┐
│ COMPREHENSIVE TOOL COMPARISON TABLE │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────┬──────────┬──────────┬──────────┬──────────┐ │
│ │ FEATURE │ BLACK │ FLAKE8 │ MYPY │ PYLINT │ │
│ ├─────────────┼──────────┼──────────┼──────────┼──────────┤ │
│ │ PRIMARY │ Format │ Style │ Types │ Design │ │
│ │ PURPOSE │ │ │ │ │ │
│ ├─────────────┼──────────┼──────────┼──────────┼──────────┤ │
│ │ AUTO-FIX │ ✅ │ ❌ │ ❌ │ ❌ │ │
│ │ │ (reformats)│ (only reports)│ (only reports)│ (only reports)│
│ ├─────────────┼──────────┼──────────┼──────────┼──────────┤ │
│ │ SPEED │ Fast │ Fast │ Medium │ Slow │ │
│ │ │ (1000+ │ (100+ │ (10-100 │ (1-10 │ │
│ │ │ files/s) │ files/s) │ files/s) │ files/s) │ │
│ ├─────────────┼──────────┼──────────┼──────────┼──────────┤ │
│ │ CONFIG │ Simple │ Simple │ Medium │ Complex │ │
│ │ COMPLEXITY │ (few │ (medium │ (many │ (many │ │
│ │ │ options) │ options) │ options) │ options) │ │
│ ├─────────────┼──────────┼──────────┼──────────┼──────────┤ │
│ │ OUTPUT │ Files │ Text │ Text │ Score │ │
│ │ │ (writes) │ (stdout) │ (stdout) │ + Report │ │
│ ├─────────────┼──────────┼──────────┼──────────┼──────────┤ │
│ │ BEST FOR │ Teams │ PEP 8 │ Type │ Code │ │
│ │ │ (no │ (style │ safety │ quality │ │
│ │ │ debates) │ police) │ (bugs) │ (design) │ │
│ ├─────────────┼──────────┼──────────┼──────────┼──────────┤ │
│ │ WHAT IT │ Code │ Style │ Type │ Design │ │
│ │ CATCHES │ style │ issues │ errors │ smells │ │
│ │ │ issues │ & simple │ before │ & │ │
│ │ │ │ bugs │ runtime │ complex │ │
│ │ │ │ │ │ issues │ │
│ ├─────────────┼──────────┼──────────┼──────────┼──────────┤ │
│ │ IDE │ Good │ Good │ Excel- │ Good │ │
│ │ INTEGRATION │ │ │ lent │ │ │
│ └─────────────┴──────────┴──────────┴──────────┴──────────┘ │
│ │
│ ⚡ USE ALL FOUR FOR COMPLETE CODE QUALITY │
└─────────────────────────────────────────────────────────────────────────────┘

3. How They Work Together​

3.1 The Development Workflow​

┌─────────────────────────────────────────────────────────────────────────────┐
│ THE COMPLETE DEVELOPMENT PIPELINE │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ 1. WRITE CODE │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ def calculate_tax(income, deductions): │ │
│ │ taxable = income - deductions │ │
│ │ return taxable * 0.2 │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │
│ 2. BLACK: AUTO-FORMAT │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ def calculate_tax(income, deductions): │ │
│ │ taxable = income - deductions │ │
│ │ return taxable * 0.2 │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ (Adds proper spacing, line breaks, etc.) │
│ │
│ 3. FLAKE8: STYLE CHECK │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ Line 1: Missing type hints │ │
│ │ Line 2: Magic number 0.2 │ │
│ │ Line 2: Hardcoded tax rate │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │
│ 4. MYPY: TYPE CHECK │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ Function missing return type annotation │ │
│ │ Parameters missing type annotations │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │
│ 5. PYLINT: DESIGN ANALYSIS │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ Score: 6.2/10 │ │
│ │ Issues: missing-docstring, magic-number, no-type-checking │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │
│ 6. IMPROVED CODE │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ def calculate_tax(income: float, deductions: float) -> float: │ │
│ │ """Calculate tax based on income and deductions.""" │ │
│ │ TAX_RATE = 0.2 │ │
│ │ taxable = income - deductions │ │
│ │ return taxable * TAX_RATE │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ Score: 9.8/10 ✅ │
└─────────────────────────────────────────────────────────────────────────────┘

3.2 Integration Order​

┌─────────────────────────────────────────────────────────────────────────────┐
│ RECOMMENDED EXECUTION ORDER │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ 1. BLACK (Formatter) │
│ ↓ Auto-fixes formatting issues │
│ │
│ 2. FLAKE8 (Linter) │
│ ↓ Checks style & simple bugs │
│ │
│ 3. MYPY (Type Checker) │
│ ↓ Validates type safety │
│ │
│ 4. PYLINT (Analyzer) │
│ ↓ Analyzes design & gives score │
│ │
│ 5. TEST │
│ ↓ Verifies functionality │
│ │
│ Why this order? │
│ • Black first: Fixes formatting so other tools focus on real issues │
│ • Flake8 before Pylint: Faster, catches obvious issues first │
│ • Mypy before Pylint: Type errors are more critical than style │
│ • Pylint last: Comprehensive but slow; run after quick checks │
└─────────────────────────────────────────────────────────────────────────────┘

4. Practical Setup for Your Tax Project​

4.1 Complete Configuration​

pyproject.toml​

[build-system]
requires = ["setuptools>=61.0", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "capital-gains"
version = "0.1.0"
description = "Brazilian capital gains tax calculator"
readme = "README.md"
requires-python = ">=3.10"
authors = [{name = "Your Name", email = "you@example.com"}]
dependencies = [
"pydantic>=2.0",
"decimal>=1.0",
]

[project.optional-dependencies]
dev = [
"pytest>=7.0",
"black>=23.0",
"flake8>=6.0",
"flake8-docstrings>=1.0",
"flake8-bugbear>=23.0",
"mypy>=1.0",
"pylint>=3.0",
"pylint-pydantic>=0.3",
"pre-commit>=3.0",
]

# Black Configuration
[tool.black]
line-length = 88
target-version = ['py311']
include = '\.pyi?$'

# isort Configuration
[tool.isort]
profile = "black"
line_length = 88

# Flake8 Configuration
[tool.flake8]
max-line-length = 88
extend-ignore = ["E203", "W503"]
exclude = [".git", "__pycache__", "build", "dist", ".venv", "venv"]
select = ["E", "F", "W", "B", "C", "D"]
max-complexity = 10

# Mypy Configuration
[tool.mypy]
python_version = "3.11"
warn_return_any = true
disallow_untyped_defs = true
disallow_incomplete_defs = true
check_untyped_defs = true
disallow_untyped_decorators = true

[[tool.mypy.overrides]]
module = "tests.*"
ignore_missing_imports = true

# Pylint Configuration
[tool.pylint]
python-version = "3.11"

[tool.pylint."MASTER"]
paths = ["src"]
load-plugins = ["pylint_pydantic"]

[tool.pylint."MESSAGES CONTROL"]
disable = [
"missing-docstring",
"too-few-public-methods",
"broad-except",
]
enable = ["design", "duplicate-code", "inconsistent-return-statements"]

[tool.pylint."DESIGN"]
max-args = 8
max-branches = 15
max-locals = 20
max-returns = 8

[tool.pylint."FORMAT"]
max-line-length = 88

4.2 Pre-commit Configuration​

.pre-commit-config.yaml​

repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.4.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-yaml
- id: check-added-large-files
- id: check-merge-conflict

- repo: https://github.com/psf/black
rev: 23.1.0
hooks:
- id: black
args: [--line-length=88]

- repo: https://github.com/pycqa/isort
rev: 5.12.0
hooks:
- id: isort
args: ["--profile", "black"]

- repo: https://github.com/pycqa/flake8
rev: 6.0.0
hooks:
- id: flake8
args: [--max-line-length=88]
additional_dependencies:
- flake8-docstrings
- flake8-bugbear
- flake8-comprehensions

- repo: https://github.com/pre-commit/mirrors-mypy
rev: v1.0.0
hooks:
- id: mypy
args: [--ignore-missing-imports]

- repo: local
hooks:
- id: pylint
name: pylint
entry: pylint
language: system
args: ["--fail-under=8.0", "src/"]
pass_filenames: false
always_run: true
stages: [commit]

4.3 Makefile for Development​

Makefile​

.PHONY: help format lint type analyze test check clean

help:
@echo "Available commands:"
@echo " format - Format code with Black and isort"
@echo " lint - Check code style with Flake8"
@echo " type - Check types with Mypy"
@echo " analyze - Analyze code with Pylint"
@echo " test - Run tests with pytest"
@echo " check - Run all checks (format, lint, type, analyze, test)"
@echo " clean - Clean generated files"

format:
@echo "Formatting code with Black..."
black --line-length 88 .
@echo "Sorting imports with isort..."
isort .

lint:
@echo "Checking code style with Flake8..."
flake8 .

type:
@echo "Checking types with Mypy..."
mypy src/ --ignore-missing-imports

analyze:
@echo "Analyzing code with Pylint..."
pylint src/ --fail-under=8.0

test:
@echo "Running tests with pytest..."
pytest -v --cov=src/capital_gains --cov-report=term-missing --cov-report=html

check: format lint type analyze test
@echo "✅ All checks passed!"

clean:
@echo "Cleaning generated files..."
rm -rf .mypy_cache .pytest_cache .coverage htmlcov build dist *.egg-info .pylint.d
find . -type d -name "__pycache__" -exec rm -rf {} + 2>/dev/null || true
find . -type f -name "*.pyc" -delete
find . -type f -name ".coverage" -delete
@echo "✅ Clean complete!"

4.4 Example Tax Code with All Tools​

"""
Tax calculation module with full type safety and quality checks.
"""

from decimal import Decimal
from functools import reduce
from typing import List, Tuple

from .models import Operation, TaxResult
from .portfolio import PortfolioState

# Constants should be uppercase with underscores
TAX_EXEMPT_THRESHOLD = Decimal("20000.00")
TAX_RATE = Decimal("0.20")


def _process_operation(
acc: Tuple[PortfolioState, List[TaxResult]],
operation: Operation,
) -> Tuple[PortfolioState, List[TaxResult]]:
"""
Process a single operation and return updated state and results.

This pure function processes buy/sell operations and maintains
portfolio state immutably.

Args:
acc: Tuple of (current PortfolioState, accumulated tax results)
operation: The buy/sell operation to process

Returns:
Tuple of (new PortfolioState, updated tax results)

Raises:
ValueError: If operation type is unknown
"""
state, results = acc

match operation.operation:
case "buy":
new_state = state.buy(operation.quantity, operation.unit_cost)
new_results = results + [TaxResult(tax=Decimal("0"))]
return new_state, new_results
case "sell":
new_state, tax = state.sell(
operation.quantity,
operation.unit_cost,
TAX_EXEMPT_THRESHOLD,
)
new_results = results + [TaxResult(tax=tax)]
return new_state, new_results
case _:
raise ValueError(f"Unknown operation type: {operation.operation}")


def calculate_taxes(operations: List[Operation]) -> List[TaxResult]:
"""
Calculate capital gains taxes for a sequence of stock operations.

This function processes operations sequentially using functional
composition (reduce pattern), maintaining immutable portfolio state.

Args:
operations: List of buy/sell operations to process in order

Returns:
List of TaxResult objects (one per operation)

Example:
>>> ops = [Operation("buy", 100, Decimal("50"))]
>>> results = calculate_taxes(ops)
>>> len(results)
1
"""
initial_state = (PortfolioState(), [])
_, results = reduce(_process_operation, operations, initial_state)
return results

What each tool checks in this code:

ToolWhat it verifiesWhy it's important for financial code
BlackConsistent formattingReadable calculations
Flake8PEP 8 compliance, no unused importsClean, maintainable logic
MypyType safety for Decimal operationsCorrect mathematical calculations
PylintFunction complexity, docstrings, designMaintainable tax logic

5. Common Issues & Solutions​

5.1 Tool Conflicts​

┌─────────────────────────────────────────────────────────────────────────────┐
│ RESOLVING TOOL CONFLICTS │
├─────────────────────────────────────────────────────────────────────────────┐
│ │
│ COMMON CONFLICT: Black vs Flake8 │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ Black formats: │
│ │ x = [1, 2, 3] │
│ │ │
│ │ Flake8 complains: │
│ │ E203 whitespace before ':' │
│ │ │
│ │ SOLUTION: Configure Flake8 to ignore Black's style │
│ │ [flake8] │
│ │ extend-ignore = E203, W503 │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ COMMON CONFLICT: Pylint vs Type Hints │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ Pylint complains: │
│ │ C0116: Missing function docstring │
│ │ │
│ │ But you're using Mypy for types and Flake8-docstrings for docs │
│ │ │
│ │ SOLUTION: Disable in Pylint config │
│ │ disable = missing-docstring │
│ └─────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘

5.2 Performance Optimization​

# Run tools efficiently
# 1. Black is fast - run it first
black src/

# 2. Flake8 with specific file patterns
flake8 --filename=*.py src/

# 3. Mypy with caching
mypy --cache-dir=.mypy_cache src/

# 4. Pylint with parallel processing
pylint -j 4 src/ # Use 4 CPU cores

# Or use pre-commit which caches results
pre-commit run --all-files

5.3 IDE Integration​

VS Code Settings​

{
"python.formatting.provider": "black",
"python.formatting.blackArgs": ["--line-length", "88"],
"python.linting.enabled": true,
"python.linting.flake8Enabled": true,
"python.linting.mypyEnabled": true,
"python.linting.pylintEnabled": true,
"python.linting.lintOnSave": true,
"editor.formatOnSave": true,
"editor.codeActionsOnSave": {
"source.organizeImports": true
},
"[python]": {
"editor.rulers": [88]
}
}

PyCharm Configuration​

  1. Black: Install "BlackConnect" plugin
  2. Flake8: Settings → Tools → Flake8
  3. Mypy: Settings → Tools → Mypy
  4. Pylint: Settings → Tools → Pylint

6. CI/CD Pipeline​

6.1 GitHub Actions Workflow​

name: Python CI

on: [push, pull_request]

jobs:
quality:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.10", "3.11"]

steps:
- uses: actions/checkout@v3

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}

- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -e ".[dev]"

- name: Format check (Black)
run: black --check --line-length 88 .

- name: Import check (isort)
run: isort --check-only --profile black .

- name: Lint (Flake8)
run: flake8 . --max-line-length=88

- name: Type check (Mypy)
run: mypy src/ --ignore-missing-imports

- name: Code analysis (Pylint)
run: pylint src/ --fail-under=8.0

- name: Run tests
run: pytest -v --cov=src/capital_gains --cov-report=xml

- name: Upload coverage to Codecov
uses: codecov/codecov-action@v3
with:
file: ./coverage.xml

6.2 GitLab CI Configuration​

stages:
- format
- lint
- type
- analyze
- test

variables:
PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"

cache:
paths:
- .cache/pip
- .mypy_cache

format:
stage: format
image: python:3.11
script:
- pip install black isort
- black --check --line-length 88 .
- isort --check-only --profile black .

lint:
stage: lint
image: python:3.11
script:
- pip install flake8 flake8-docstrings flake8-bugbear
- flake8 . --max-line-length=88

type-check:
stage: type
image: python:3.11
script:
- pip install mypy
- mypy src/ --ignore-missing-imports

analyze:
stage: analyze
image: python:3.11
script:
- pip install pylint
- pylint src/ --fail-under=8.0

test:
stage: test
image: python:3.11
script:
- pip install pytest pytest-cov
- pytest -v --cov=src/capital_gains --cov-report=term-missing

7. Best Practices for Financial Code​

7.1 Financial-Specific Configuration​

# .flake8 (financial adjustments)
[flake8]
max-line-length = 88
# Allow more complex calculations
max-complexity = 15 # Increased from default 10
# Financial functions may need more parameters
per-file-ignores =
financial_calculations.py:WPS231 # Allow more complex calculations
tax_engine.py:R0913 # Allow more arguments in tax functions

# mypy.ini (strict for financials)
[mypy]
strict = true # Enable all checks
disallow_any_unimported = true
disallow_any_expr = false # Allow for dynamic calculations
warn_return_any = true
warn_unused_ignores = true

# Financial modules get extra strictness
[mypy.capital_gains.calculator]
disallow_untyped_defs = true
disallow_incomplete_defs = true

7.2 Example: Financial Code Quality​

"""
High-quality financial code leveraging all tools.
"""

from decimal import Decimal, ROUND_HALF_UP
from typing import List, Optional


class TaxCalculator:
"""Calculate taxes with full type safety and error handling."""

STANDARD_DEDUCTION = Decimal("13850.00")
TAX_BRACKETS = [
(Decimal("0.00"), Decimal("0.10")),
(Decimal("11000.00"), Decimal("0.12")),
(Decimal("44725.00"), Decimal("0.22")),
(Decimal("95375.00"), Decimal("0.24")),
]

def __init__(self, filing_status: str = "single") -> None:
"""Initialize tax calculator with filing status."""
self.filing_status = filing_status
self.validate_status()

def validate_status(self) -> None:
"""Validate filing status."""
valid_statuses = {"single", "married_joint", "married_separate", "head"}
if self.filing_status not in valid_statuses:
raise ValueError(f"Invalid filing status: {self.filing_status}")

def calculate_tax(
self,
income: Decimal,
deductions: Optional[Decimal] = None,
credits: Decimal = Decimal("0.00"),
) -> Decimal:
"""
Calculate tax liability.

Args:
income: Gross income
deductions: Optional itemized deductions
credits: Tax credits to apply

Returns:
Tax liability rounded to nearest cent

Raises:
ValueError: If income is negative
"""
if income < Decimal("0.00"):
raise ValueError("Income cannot be negative")

# Use standard deduction if none provided
actual_deductions = deductions or self.STANDARD_DEDUCTION

# Calculate taxable income
taxable_income = max(income - actual_deductions, Decimal("0.00"))

# Calculate tax using brackets
tax = Decimal("0.00")
previous_bracket = Decimal("0.00")

for bracket, rate in self.TAX_BRACKETS:
if taxable_income > bracket:
bracket_amount = min(taxable_income, bracket) - previous_bracket
tax += bracket_amount * rate
previous_bracket = bracket
else:
break

# Apply credits and round
final_tax = max(tax - credits, Decimal("0.00"))
return final_tax.quantize(Decimal("0.01"), rounding=ROUND_HALF_UP)

8. Troubleshooting Guide​

┌─────────────────────────────────────────────────────────────────────────────┐
│ TROUBLESHOOTING COMMON ISSUES │
├─────────────────────────────────────────────────────────────────────────────┐
│ │
│ ISSUE: Tools reporting different errors │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ Problem: │ │
│ │ • Black formats one way │ │
│ │ • Flake8 expects another │ │
│ │ │
│ │ Solution: Align configurations │ │
│ │ 1. Black line-length: 88 │ │
│ │ 2. Flake8 max-line-length: 88 │ │
│ │ 3. Pylint max-line-length: 88 │ │
│ │ 4. Flake8 ignore: E203, W503 (Black conflicts) │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ISSUE: Slow performance on large codebase │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ Solutions: │ │
│ │ 1. Use caching: │ │
│ │ mypy --cache-dir=.mypy_cache │ │
│ │ 2. Parallel execution: │ │
│ │ pylint -j 4 │ │
│ │ 3. Run only on changed files: │ │
│ │ pre-commit run --files changed_file.py │ │
│ │ 4. Exclude generated files: │
│ │ exclude = __pycache__, build, dist │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ISSUE: False positives in tests or migrations │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ Solutions: │ │
│ │ 1. Per-file ignores: │ │
│ │ [flake8] │ │
│ │ per-file-ignores = tests/*:S101 │ │
│ │ 2. Directory-specific configs: │ │
│ │ [mypy-tests.*] │ │
│ │ ignore_missing_imports = true │ │
│ │ 3. Inline disables: │ │
│ │ # pylint: disable=protected-access │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘

9. Quick Reference Cheat Sheet​

# INSTALL ALL TOOLS
pip install black flake8 mypy pylint

# BASIC USAGE
black src/ # Format code
flake8 src/ # Check style
mypy src/ # Check types
pylint src/ # Analyze code

# WITH COMMON OPTIONS
black --line-length 88 src/
flake8 --max-line-length 88 src/
mypy --ignore-missing-imports src/
pylint --fail-under 8.0 src/

# CHECK WITHOUT FIXING
black --check src/
flake8 src/ # Always check-only
mypy src/ # Always check-only
pylint src/ # Always check-only

# INTEGRATED WORKFLOW
make check # Run all checks
pre-commit run --all-files # Run pre-commit hooks

10. Conclusion: The Complete Quality Stack​

Final Recommendation for Your Tax Project:​

"""
IMPLEMENT THIS WORKFLOW FOR YOUR CAPITAL GAINS PROJECT:
"""

# 1. SETUP (One-time)
# --------------------
# Create configuration files:
# - pyproject.toml (Black, isort, Flake8, Mypy, Pylint config)
# - .pre-commit-config.yaml (Git hooks)
# - Makefile (Development commands)

# 2. DEVELOPMENT (Daily)
# ----------------------
# Workflow:
# 1. Write code with type hints
# 2. Run `make format` (Black + isort)
# 3. Run `make check` (All quality checks)
# 4. Fix any issues
# 5. Commit (pre-commit hooks run automatically)

# 3. CI/CD (Automated)
# --------------------
# Every commit/pull request runs:
# - Black formatting check
# - Flake8 style check
# - Mypy type check
# - Pylint analysis
# - pytest tests
# - Code coverage

# 4. BENEFITS FOR FINANCIAL CODE
# ------------------------------
# • Type safety = Fewer calculation errors
# • Clean code = Understandable tax logic
# • Good design = Maintainable financial rules
# • Consistency = Team collaboration

The Tool Roles Recap:​

ToolRoleWhy Essential for Financial Code
BlackAuto-formatterSaves time, ensures consistency in complex calculations
Flake8Style enforcerMakes financial logic readable and maintainable
MypyType checkerCatches Decimal/float errors before they cause financial miscalculations
PylintCode analyzerEnsures tax calculation code is well-designed and not overly complex

Start Simple, Then Scale:​

  1. Week 1: Add Black for formatting
  2. Week 2: Add Flake8 for basic linting
  3. Week 3: Add Mypy for type safety
  4. Week 4: Add Pylint for design analysis
  5. Week 5: Add pre-commit hooks
  6. Week 6: Add CI/CD pipeline

Each tool adds a layer of quality assurance. Together, they create a defense-in-depth strategy for code quality that's especially important for financial applications where correctness is paramount.

Remember: The goal isn't perfection, but continuous improvement. Start with basic configurations, tighten them as your team gets comfortable, and enjoy the benefits of professional, reliable Python code for your capital gains tax calculations.

The reduce Function: A Comprehensive Guide

· 11 min read

1. Core Concept​

The reduce function (also known as fold, accumulate, or aggregate) is a higher-order function that processes elements of an iterable and reduces them to a single value by repeatedly applying a binary function. It works like an assembly line that processes items one by one, accumulating results as it goes.

Visual Metaphor: Assembly Line Factory​

┌───────────────────────────────────────────────────────────────────────┐
│ REDUCE ASSEMBLY LINE │
├───────────────────────────────────────────────────────────────────────┤
│ │
│ Input Items: [🍎, 🍌, 🍒, 🍇] │
│ Processor: "Add to basket" function │
│ Starting Point: Empty basket [] │
│ │
│ Step 1: Empty basket + 🍎 = [🍎] │
│ Step 2: Basket [🍎] + 🍌 = [🍎, 🍌] │
│ Step 3: Basket [🍎, 🍌] + 🍒 = [🍎, 🍌, 🍒] │
│ Step 4: Basket [🍎, 🍌, 🍒] + 🍇 = [🍎, 🍌, 🍒, 🍇] │
│ │
│ Final Output: Basket with all fruits [🍎, 🍌, 🍒, 🍇] │
│ │
└───────────────────────────────────────────────────────────────────────┘

2. Python Syntax​

from functools import reduce

# Basic syntax
reduce(function, iterable, initializer=None)

3. Understanding the Three Components​

Component 1: The Processing Function​

This is a binary function that takes two arguments:

  • First argument: The current accumulated value (starts with initializer)
  • Second argument: The next element from the iterable

The function returns a new accumulated value that will be used in the next iteration.

# Example processing function for summing numbers
def add(current_total, next_number):
return current_total + next_number

# The same as lambda: lambda acc, x: acc + x

Component 2: The Iterable​

Any Python object that can be looped over:

  • Lists: [1, 2, 3, 4]
  • Tuples: ('a', 'b', 'c')
  • Strings: "hello" (iterates over characters)
  • Generators: (x for x in range(5))
  • And more...

Component 3: The Initializer (Optional)​

The starting value for the accumulation. If not provided:

  • The first element of the iterable becomes the initial accumulator value
  • Processing starts from the second element
# With initializer
total = reduce(lambda acc, x: acc + x, [1, 2, 3], 0) # Starts with 0

# Without initializer
total = reduce(lambda acc, x: acc + x, [1, 2, 3]) # Starts with 1

4. How Reduce Works: Step-by-Step Visualization​

Processing Flow Diagram​

┌─────────────────────────────────────────────────────────────────────────┐
│ REDUCE PROCESS FLOW │
├─────────┬──────────────┬──────────────┬──────────────┬──────────────────┤
│ Step │ Accumulator │ Current Item │ Function │ New Value │
│ │ (Start Value)│ (From Iter.) │ Applied │ Becomes Next │
│ │ │ │ │ Accumulator │
├─────────┼──────────────┼──────────────┼──────────────┼──────────────────┤
│ Start │ initial_val │ - │ - │ initial_val │
│ │ │ │ │ │
│ Step 1 │ initial_val │ items[0] │ func(initial │ result₁ → │
│ │ │ │ _val, │ Next Accumulator │
│ │ │ │ items[0]) │ │
│ │ │ │ │ │
│ Step 2 │ result₁ │ items[1] │ func(result₁│ result₂ → │
│ │ │ │ , items[1]) │ Next Accumulator │
│ │ │ │ │ │
│ Step 3 │ result₂ │ items[2] │ func(result₂│ result₃ → │
│ │ │ │ , items[2]) │ Next Accumulator │
│ │ │ │ │ │
│ ... │ ... │ ... │ ... │ ... │
│ │ │ │ │ │
│ Step N │ resultₙ₋₁ │ items[n-1] │ func(resultₙ│ Final Result │
│ │ │ │ ₋₁, items[n-1]) │
└─────────┴──────────────┴──────────────┴──────────────┴──────────────────┘

Simple Example: Summation Process​

numbers = [1, 2, 3, 4, 5]

total = reduce(lambda acc, x: acc + x, numbers, 0)

# Visualizing each step:
"""
Step 0: Start with accumulator = 0 (initializer)
Step 1: accumulator = 0, x = 1 → 0 + 1 = 1 (new accumulator)
Step 2: accumulator = 1, x = 2 → 1 + 2 = 3 (new accumulator)
Step 3: accumulator = 3, x = 3 → 3 + 3 = 6 (new accumulator)
Step 4: accumulator = 6, x = 4 → 6 + 4 = 10 (new accumulator)
Step 5: accumulator = 10, x = 5 → 10 + 5 = 15 (final result)
"""

5. Key Characteristics​

1. Sequential Processing​

Items are processed in order, from first to last. Each step builds upon the previous result.

2. Stateful Accumulation​

The accumulator maintains state across iterations. This makes reduce ideal for problems where each step depends on previous steps.

3. Pure Function Ideal​

While not required, reduce works best with pure functions (no side effects) for predictable behavior.

4. Flexible Return Type​

The accumulator can be any data type:

  • Numbers (for sums, products)
  • Lists (for flattening, filtering)
  • Dictionaries (for counting, grouping)
  • Custom objects (for complex state management)

6. The Processing Function: Deep Dive​

Function Signature Requirements​

The processing function must accept exactly two parameters:

def processing_function(accumulator, current_item):
# accumulator: Current accumulated value
# current_item: Next element from the iterable
# Return: New accumulated value
return new_accumulator_value

Common Processing Patterns​

# Pattern 1: Aggregation (sum, product)
sum_func = lambda acc, x: acc + x
product_func = lambda acc, x: acc * x

# Pattern 2: Collection (building lists, strings)
collect_func = lambda acc, x: acc + [x] # Build list
concat_func = lambda acc, s: acc + s # Concatenate strings

# Pattern 3: Transformation (with state)
running_avg = lambda acc, x: (acc[0] + x, acc[1] + 1) # Track total and count

# Pattern 4: Filtering (conditional accumulation)
filter_even = lambda acc, x: acc + [x] if x % 2 == 0 else acc

7. Initializer: When to Use It​

Case 1: Initializer Required​

When the result type differs from the input type:

# Counting occurrences (input: strings, output: dict)
words = ["apple", "banana", "apple", "orange"]
word_count = reduce(
lambda acc, word: {**acc, word: acc.get(word, 0) + 1},
words,
{} # Initializer is empty dict
)

Case 2: Initializer Optional​

When the operation is associative and starting value is clear:

# Summation can start with 0 or first element
numbers = [1, 2, 3, 4]

# With initializer (explicit start)
sum1 = reduce(lambda acc, x: acc + x, numbers, 0) # 10

# Without initializer (uses first element as start)
sum2 = reduce(lambda acc, x: acc + x, numbers) # Also 10

Case 3: Special Initial Values​

When the operation requires a specific starting point:

# Find maximum with float('-inf') as starting point
numbers = [-5, -10, -3]
max_val = reduce(lambda acc, x: x if x > acc else acc, numbers, float('-inf'))

# String concatenation starting with prefix
strings = ["world", "!"]
result = reduce(lambda acc, s: acc + " " + s, strings, "Hello") # "Hello world !"

8. Basic Examples​

Example 1: Product of Numbers​

numbers = [2, 3, 4, 5]

product = reduce(lambda acc, x: acc * x, numbers, 1)
# Step-by-step: 1 * 2 = 2, 2 * 3 = 6, 6 * 4 = 24, 24 * 5 = 120
# Result: 120

Example 2: Find Maximum Value​

numbers = [3, 7, 2, 9, 4]

max_value = reduce(lambda acc, x: x if x > acc else acc, numbers, numbers[0])
# Step-by-step: 3 vs 7 → 7, 7 vs 2 → 7, 7 vs 9 → 9, 9 vs 4 → 9
# Result: 9

Example 3: String Concatenation​

words = ["Hello", " ", "World", "!"]

result = reduce(lambda acc, word: acc + word, words, "")
# Step-by-step: "" + "Hello" = "Hello"
# "Hello" + " " = "Hello "
# "Hello " + "World" = "Hello World"
# "Hello World" + "!" = "Hello World!"
# Result: "Hello World!"

9. Intermediate Examples​

Example 4: Flatten Nested Lists​

nested_lists = [[1, 2], [3, 4, 5], [6], [7, 8, 9]]

flattened = reduce(lambda acc, lst: acc + lst, nested_lists, [])
# Step 1: [] + [1, 2] = [1, 2]
# Step 2: [1, 2] + [3, 4, 5] = [1, 2, 3, 4, 5]
# Step 3: [1, 2, 3, 4, 5] + [6] = [1, 2, 3, 4, 5, 6]
# Step 4: [1, 2, 3, 4, 5, 6] + [7, 8, 9] = [1, 2, 3, 4, 5, 6, 7, 8, 9]
# Result: [1, 2, 3, 4, 5, 6, 7, 8, 9]

Example 5: Count Word Frequencies​

words = ["apple", "banana", "apple", "orange", "banana", "apple"]

word_counts = reduce(
lambda acc, word: {**acc, word: acc.get(word, 0) + 1},
words,
{}
)
# Step 1: {} + "apple" → {"apple": 1}
# Step 2: {"apple": 1} + "banana" → {"apple": 1, "banana": 1}
# Step 3: {"apple": 1, "banana": 1} + "apple" → {"apple": 2, "banana": 1}
# Step 4: {"apple": 2, "banana": 1} + "orange" → {"apple": 2, "banana": 1, "orange": 1}
# Step 5: {"apple": 2, "banana": 1, "orange": 1} + "banana" → {"apple": 2, "banana": 2, "orange": 1}
# Step 6: {"apple": 2, "banana": 2, "orange": 1} + "apple" → {"apple": 3, "banana": 2, "orange": 1}
# Result: {'apple': 3, 'banana': 2, 'orange': 1}

Example 6: Running Average Calculation​

numbers = [10, 20, 30, 40, 50]

# Track both total and count
final_total, final_count, averages = reduce(
lambda acc, x: (
acc[0] + x, # New total
acc[1] + 1, # New count
acc[2] + [(acc[0] + x) / (acc[1] + 1)] # New average list
),
numbers,
(0, 0, []) # Initial: (total, count, averages_list)
)

# averages contains: [10.0, 15.0, 20.0, 25.0, 30.0]

10. Advanced Examples​

Example 7: State Machine Processing​

# Process a sequence of events with state transitions
events = ["START", "PROCESS", "PROCESS", "END"]
initial_state = "IDLE"

def state_transition(current_state, event):
transitions = {
("IDLE", "START"): "RUNNING",
("RUNNING", "PROCESS"): "RUNNING",
("RUNNING", "END"): "IDLE",
}
return transitions.get((current_state, event), "ERROR")

final_state = reduce(state_transition, events, initial_state)
# Result: "IDLE" (after processing all events)

Example 8: Pipeline Composition​

# Compose multiple functions into a single pipeline
def add_5(x): return x + 5
def multiply_2(x): return x * 2
def subtract_3(x): return x - 3

functions = [add_5, multiply_2, subtract_3]

# Create composed function: subtract_3(multiply_2(add_5(x)))
composed = reduce(lambda f, g: lambda x: g(f(x)), functions)

result = composed(10) # ((10 + 5) * 2) - 3 = 27

Example 9: Matrix Operations​

matrix = [
[1, 2, 3],
[4, 5, 6],
[7, 8, 9]
]

# Calculate column-wise sums
column_sums = reduce(
lambda acc, row: [acc[i] + val for i, val in enumerate(row)],
matrix,
[0, 0, 0] # Initialize with three zeros for three columns
)
# Step 1: [0,0,0] + [1,2,3] → [1,2,3]
# Step 2: [1,2,3] + [4,5,6] → [5,7,9]
# Step 3: [5,7,9] + [7,8,9] → [12,15,18]
# Result: [12, 15, 18]

11. Real-World Application: Tax Calculation System​

The following example demonstrates how reduce can manage complex state transitions in a financial application:

from decimal import Decimal
from functools import reduce

def calculate_taxes(operations):
"""
Process stock operations sequentially, maintaining portfolio state
and calculating taxes for each sell operation.

This demonstrates reduce's power for stateful sequential processing.
"""

def process_operation(acc, operation):
"""
State transition function for processing one operation.

Args:
acc: Tuple of (portfolio_state, tax_results_so_far)
operation: Current operation to process

Returns:
Tuple of (updated_state, updated_results)
"""
state, results = acc

if operation.type == "buy":
# Update portfolio with purchased shares
new_state = state.add_shares(operation.quantity, operation.price)
# Buy operations have zero tax
new_results = results + [TaxResult(tax=Decimal("0"))]
return new_state, new_results

elif operation.type == "sell":
# Calculate profit/loss and tax
new_state, tax_due = state.sell_shares(
operation.quantity,
operation.price,
tax_exempt_threshold=Decimal("20000.00")
)
new_results = results + [TaxResult(tax=tax_due)]
return new_state, new_results

# Start with empty portfolio and no results
initial_state = (Portfolio(), [])

# Process all operations sequentially
_, tax_results = reduce(process_operation, operations, initial_state)

return tax_results

Visualization of Tax Calculation Process:​

┌─────────────────────────────────────────────────────────────────────────┐
│ TAX CALCULATION PIPELINE │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ Operations: [Buy, Sell, Buy, Sell, Sell] │
│ Initial State: Empty Portfolio │
│ │
│ reduce() processes: │
│ 1. Takes initial state (empty portfolio) │
│ 2. Applies process_operation to first buy operation │
│ 3. Gets new state (portfolio with shares) │
│ 4. Applies process_operation to next sell operation │
│ 5. Continues through all operations │
│ 6. Returns final state and all tax results │
│ │
│ Key Insight: Each step receives the complete state from │
│ previous steps, enabling complex cumulative calculations │
│ like weighted averages, loss carryforwards, and tax exemptions. │
└─────────────────────────────────────────────────────────────────────────┘

12. When to Use Reduce​

Good Use Cases:​

  • Sequential state processing: Where each step depends on previous results
  • Cumulative calculations: Running totals, averages, aggregates
  • Building complex objects: Gradual construction of dictionaries, lists, or custom objects
  • Pipeline processing: Applying multiple transformations in sequence
  • Functional programming patterns: When working with pure functions

When to Consider Alternatives:​

  • Simple accumulations: Use built-ins like sum(), max(), min()
  • Readability concerns: Sometimes explicit loops are clearer
  • Very large datasets: Consider generators or specialized libraries
  • Parallel processing: reduce is inherently sequential

13. Performance Considerations​

# reduce vs loop performance comparison
from timeit import timeit

data = list(range(10000))

def sum_with_loop(data):
total = 0
for x in data:
total += x
return total

def sum_with_reduce(data):
return reduce(lambda acc, x: acc + x, data, 0)

# Performance is typically similar
# Choose based on readability and context

14. Common Pitfalls and Best Practices​

Pitfall 1: Missing Initializer​

# Problem: Empty iterable without initializer raises TypeError
result = reduce(lambda acc, x: acc + x, []) # TypeError!

# Solution: Provide initializer
result = reduce(lambda acc, x: acc + x, [], 0) # Returns 0

Pitfall 2: Mutable Default Arguments​

# Problem: Unexpected behavior with mutable accumulators
# Don't do this:
result = reduce(lambda acc, x: acc.append(x) or acc, [1, 2, 3], [])

# Solution: Create new objects
result = reduce(lambda acc, x: acc + [x], [1, 2, 3], [])

Best Practice 1: Use Named Functions for Complexity​

# Instead of complex lambdas:
result = reduce(lambda acc, x: complicated_expression, data, start)

# Use named functions:
def process_item(accumulator, item):
# Clear logic with comments
return new_accumulator

result = reduce(process_item, data, start)

Best Practice 2: Consider Readability​

# Sometimes a loop is clearer:
def calculate_total(items):
total = 0
for item in items:
total += item.price * item.quantity
return total

# vs reduce version:
def calculate_total(items):
return reduce(
lambda acc, item: acc + (item.price * item.quantity),
items,
0
)

15. Comparison with Other Functional Tools​

┌─────────────────────────────────────────────────────────────────────────┐
│ FUNCTIONAL PROGRAMMING TOOLKIT │
├─────────────┬──────────────────┬──────────────────┬─────────────────────┤
│ Tool │ Input │ Process │ Output │
├─────────────┼──────────────────┼──────────────────┼─────────────────────┤
│ map() │ [a, b, c] │ Apply function │ [f(a), f(b), f(c)] │
│ │ │ to each element │ │
│ │ │ │ │
│ filter() │ [a, b, c] │ Keep elements │ [a, c] if f(a) and │
│ │ │ where f(x)=True │ f(c) are True │
│ │ │ │ │
│ reduce() │ [a, b, c] │ Apply binary │ f(f(start,a),b),c) │
│ │ │ function │ │
│ │ │ cumulatively │ │
└─────────────┴──────────────────┴──────────────────┴─────────────────────┘

16. Summary​

The reduce function is a powerful tool for sequential data processing that:

  1. Processes items one by one in a defined order
  2. Maintains state across iterations through the accumulator
  3. Transforms an iterable into a single value (which can be complex)
  4. Works with any data type as both input and output
  5. Enables pure functional patterns with predictable behavior

The key insight is that reduce provides a pattern for stateful iteration where each step has access to the complete results of all previous steps. This makes it particularly valuable for:

  • Financial calculations (like the tax example)
  • Data aggregation and summarization
  • Complex object construction
  • Sequential decision making
  • Pipeline and workflow processing

While not always the most readable choice for simple tasks, reduce excels at problems where maintaining and transforming state across multiple steps is essential to the solution.

Python -m Flag Explained: Module Mode Execution

· 5 min read

1. Basic Concept​

python -m means run Python in module mode. The -m stands for module.

Syntax:​

python -m module_name [args...]

2. -m vs Direct Script Execution​

Example 1: Direct script execution​

# Method 1: Direct run (relative path)
python my_script.py

# Method 2: Direct run (absolute path)
python /path/to/my_script.py

Example 2: Using -m​

# Method 3: Run as module
python -m my_package.my_module

3. Key Difference: sys.path Behavior​

This is the most important feature of -m! Let's demonstrate with code:

Create test directory structure:​

test_project/
├── my_package/
│ ├── __init__.py
│ └── my_module.py
└── main.py

Test code:​

my_package/my_module.py:

import sys

def show_path():
print("Module path:", __file__)
print("First in sys.path:", sys.path[0])
print("-" * 40)

main.py:

import sys
print("=== Direct execution ===")
print("Script path:", __file__)
print("First in sys.path:", sys.path[0])

print("\n=== Now importing module ===")
from my_package import my_module
my_module.show_path()

Execution comparison:​

# Run from current directory (test_project/)

# Method 1: Direct execution
$ python main.py
=== Direct execution ===
Script path: /path/to/test_project/main.py
First in sys.path: /path/to/test_project # Current directory first

=== Now importing module ===
Module path: /path/to/test_project/my_package/my_module.py
First in sys.path: /path/to/test_project

# Method 2: Using -m
$ python -m my_package.my_module
Module path: /path/to/test_project/my_package/my_module.py
First in sys.path: '' # Empty string! Means current working directory

4. Unique Features of -m​

4.1 Automatic current directory in sys.path​

# test.py
import sys
print("Current directory in sys.path:", os.getcwd() in sys.path)
$ python test.py
Current directory in sys.path: True # Always in sys.path

$ python -m test
Current directory in sys.path: True # Also in sys.path, but different position

4.2 Proper package structure handling​

myapp/
├── __init__.py
├── utils/
│ ├── __init__.py
│ └── helpers.py
└── main.py

Wrong way:

$ cd myapp
$ python main.py # May fail to import utils.helpers correctly

Correct way:

$ cd ..
$ python -m myapp.main # Ensures proper package resolution

5. Common Use Cases​

5.1 Running standard library modules​

# Start simple HTTP server
python -m http.server 8000

# Run JSON tool
echo '{"name": "Alice"}' | python -m json.tool

# Code profiling
python -m cProfile my_script.py

# Debugging
python -m pdb my_script.py

# Packaging tools
python -m pip install requests
python -m venv myenv # Create virtual environment
python -m ensurepip # Install pip

5.2 Running modules within packages​

# Assume package structure:
# mypackage/
# ├── __init__.py
# ├── cli.py
# └── utils.py

# Can run from anywhere
python -m mypackage.cli --help

5.3 Development testing​

# Run tests
python -m pytest tests/
python -m unittest discover

# Run development servers
python -m flask run
python -m uvicorn main:app --reload

6. Special Role of __main__.py​

When using -m with a package, Python looks for __main__.py:

Example:​

myapp/
├── __init__.py
├── __main__.py # Package entry point
└── other_files.py

myapp/main.py:

def main():
print("Hello from myapp!")

if __name__ == "__main__":
main()

Execution:

# Run package's __main__.py
python -m myapp
# Output: Hello from myapp!

# Equivalent to
python myapp/__main__.py
# But -m handles imports better

7. Practical Comparison​

Scenario: Relative imports within packages​

Directory structure:

mylib/
├── __init__.py
├── math/
│ ├── __init__.py
│ └── operations.py # Contains: from . import advanced
├── advanced.py
└── runner.py

operations.py:

# Using relative import
from . import advanced

def add(x, y):
return x + y + advanced.BONUS

Try to run:

# ❌ Method 1: Direct run (fails)
$ cd mylib/math
$ python operations.py
# ImportError: attempted relative import with no known parent package

# ❌ Method 2: Run from parent (also fails)
$ cd mylib
$ python math/operations.py
# ImportError: attempted relative import with no known parent package

# ✅ Method 3: Using -m (succeeds!)
$ cd mylib
$ python -m math.operations
# Or from anywhere:
$ python -m mylib.math.operations

8. Technical Details of -m​

What Python actually does:​

python -m module.name
  1. Resolve module path: Convert module.name to filesystem path
  2. Set sys.path[0] to "" (empty string), representing current working directory
  3. Execute module: Like import module.name then running its code
  4. Handle __main__: Set module's __name__ to "__main__"

See how Python finds modules:​

# View module search locations
python -m site

# Find module file location
python -m module_name --help # Many modules support --help
python -c "import module_name; print(module_name.__file__)"

9. Comparison with python -c​

# -m: Run module
python -m http.server

# -c: Execute code string directly
python -c "print('Hello')"

# Combination (not actually valid, showing contrast)
python -c "import sys; print(sys.path)" # View current path

10. Common Problems & Solutions​

Q1: Why do my relative imports fail with direct execution but work with -m?​

A: Direct execution doesn't know your file belongs to a package; -m provides proper package context.

Q2: What's the difference between python script.py and python -m script?​

# Assume script.py is in current directory

# Method 1: Direct execution
python script.py
# sys.path[0] = script's directory

# Method 2: Module execution
python -m script
# sys.path[0] = current working directory (may differ!)

Q3: When must I use -m?​

  1. Running modules within packages (with relative imports)
  2. Running standard library tools (like pip, venv)
  3. Ensuring consistent import behavior

11. Best Practices​

  1. Project entry points: Always use python -m myproject.module
  2. Executable packages: Add __main__.py to your packages
  3. Development tools: Use python -m pytest instead of global pytest
  4. Virtual environments: Use python -m venv instead of virtualenv

Example project structure:​

myapp/
├── pyproject.toml
├── src/
│ └── myapp/
│ ├── __init__.py
│ ├── __main__.py
│ └── cli.py
└── tests/

Execution:

# During development
cd myapp
python -m pip install -e .
python -m myapp --help

# For end users
python -m pip install myapp
python -m myapp --version

12. Summary​

Core advantages of python -m:

FeatureDirect execution (python script.py)Module execution (python -m module)
sys.path[0]Script's directoryCurrent working directory ("")
Package contextNoneYes (supports relative imports)
Stdlib toolsCannot run directlyCan run directly
Cross-directoryNeeds full pathCan run from anywhere
ConsistencyDepends on execution locationMore consistent import behavior

Simple mnemonic:

  • -m = module (module mode)
  • Solves relative import and package structure issues
  • Provides consistent execution environment
  • Recommended for modern Python projects

When you need consistent import behavior, especially for modules within packages, always prefer python -m!

Deep Dive: Python Immutable Data Classes

· 12 min read

📋 Table of Contents​

  1. Introduction & Fundamentals
  2. How It Works Internally
  3. Practical Benefits & Use Cases
  4. The Shallow Freezing Caveat
  5. Working with Immutable Objects
  6. Advanced Patterns & Alternatives
  7. Performance Considerations
  8. Best Practices

Introduction & Fundamentals​

What is @dataclass(frozen=True)?​

The @dataclass(frozen=True) decorator in Python creates immutable data classes - classes primarily designed for storing data while enforcing strict immutability on their instances. Introduced in Python 3.7's dataclasses module, it combines automatic method generation with functional programming principles.

Core Concept: Immutability vs. Mutability​

# Mutable class (standard dataclass)
@dataclass
class MutablePoint:
x: float
y: float

p1 = MutablePoint(1, 2)
p1.x = 3 # ✅ Modification allowed

# Immutable class (frozen dataclass)
@dataclass(frozen=True)
class ImmutablePoint:
x: float
y: float

p2 = ImmutablePoint(1, 2)
p2.x = 3 # ❌ Raises FrozenInstanceError

How It Works Internally​

Automatic Method Generation​

When you use @dataclass(frozen=True), Python automatically generates these methods:

MethodPurposeGenerated With frozen=True
__init__Constructor✅ Yes
__repr__String representation✅ Yes
__eq__Equality comparison✅ Yes
__hash__Hashing support✅ Yes (crucial!)
__setattr__Attribute assignment✅ Modified to raise error
__delattr__Attribute deletion✅ Modified to raise error
__lt__, __le__, etc.Comparisons✅ If order=True

The Magic of __hash__​

The most important addition is __hash__:

@dataclass(frozen=True)
class User:
id: int
name: str

# Objects are hashable and can be used in sets/dicts
users_set = {User(1, "Alice"), User(2, "Bob")}
users_dict = {User(1, "Alice"): "admin", User(2, "Bob"): "user"}

Key Insight: Without frozen=True, dataclasses are unhashable by default unless you explicitly set unsafe_hash=True. With frozen=True, hashability comes for free.

Implementation Example​

Here's what Python essentially creates under the hood:

class ImmutablePoint:
def __init__(self, x: float, y: float):
object.__setattr__(self, 'x', x)
object.__setattr__(self, 'y', y)

def __setattr__(self, name, value):
raise FrozenInstanceError(f"cannot assign to field '{name}'")

def __delattr__(self, name):
raise FrozenInstanceError(f"cannot delete field '{name}'")

def __repr__(self):
return f"ImmutablePoint(x={self.x}, y={self.y})"

def __eq__(self, other):
if not isinstance(other, ImmutablePoint):
return NotImplemented
return self.x == other.x and self.y == other.y

def __hash__(self):
return hash((self.x, self.y))

Practical Benefits & Use Cases​

1. Thread Safety & Concurrency​

from threading import Thread
from dataclasses import dataclass
import time

@dataclass(frozen=True)
class SharedConfig:
max_connections: int
timeout: int
api_key: str

config = SharedConfig(max_connections=100, timeout=30, api_key="secret")

# Multiple threads can safely read without locks
def worker(config: SharedConfig, thread_id: int):
for _ in range(3):
print(f"Thread {thread_id}: Max connections = {config.max_connections}")
time.sleep(0.1)

threads = [Thread(target=worker, args=(config, i)) for i in range(5)]
for t in threads:
t.start()
for t in threads:
t.join()

2. Dictionary Keys & Set Elements​

@dataclass(frozen=True)
class Coordinate:
x: int
y: int
z: int = 0

# Create a cache of computed values
cache = {}

def compute_value(coord: Coordinate) -> float:
if coord in cache: # ✅ Can use as dict key
return cache[coord]

# Expensive computation
result = (coord.x ** 2 + coord.y ** 2 + coord.z ** 2) ** 0.5
cache[coord] = result
return result

# Create a set of unique coordinates
unique_points = {
Coordinate(1, 2),
Coordinate(1, 2), # Duplicate - will be ignored
Coordinate(3, 4),
Coordinate(1, 2, 5), # Different because z=5
}
print(f"Unique points: {len(unique_points)}") # 3

3. Functional Programming Patterns​

from dataclasses import dataclass
from typing import List

@dataclass(frozen=True)
class ShoppingCart:
items: tuple[str, ...] # Use tuple for immutability
discount: float = 0.0

def add_item(self, item: str) -> 'ShoppingCart':
"""Return a NEW cart with the added item"""
from dataclasses import replace
return replace(self, items=self.items + (item,))

def apply_discount(self, new_discount: float) -> 'ShoppingCart':
"""Return a NEW cart with updated discount"""
from dataclasses import replace
return replace(self, discount=new_discount)

# Original cart remains unchanged
cart1 = ShoppingCart(items=("apple", "banana"))
cart2 = cart1.add_item("orange")
cart3 = cart2.apply_discount(0.1)

print(f"Cart1: {cart1}") # Items: ('apple', 'banana'), Discount: 0.0
print(f"Cart2: {cart2}") # Items: ('apple', 'banana', 'orange'), Discount: 0.0
print(f"Cart3: {cart3}") # Items: ('apple', 'banana', 'orange'), Discount: 0.1

4. Configuration Objects​

@dataclass(frozen=True)
class DatabaseConfig:
host: str
port: int
database: str
username: str
password: str # In reality, use a proper secrets manager

@property
def connection_string(self) -> str:
return f"postgresql://{self.username}:{self.password}@{self.host}:{self.port}/{self.database}"

# Safe to pass around - no accidental modifications
config = DatabaseConfig(
host="localhost",
port=5432,
database="mydb",
username="admin",
password="secret123"
)

# Can be safely stored in a module-level constant
DEFAULT_CONFIG = DatabaseConfig(
host="prod-db.example.com",
port=5432,
database="production",
username="readonly",
password="readonly123"
)

The Shallow Freezing Caveat​

Understanding the Limitation​

from dataclasses import dataclass

@dataclass(frozen=True)
class Company:
name: str
employees: list[str] # 🚨 Mutable type!

# Create an instance
google = Company("Google", ["Alice", "Bob"])

# This is blocked (good):
# google.name = "Alphabet" # ❌ FrozenInstanceError

# This still works (problematic):
google.employees.append("Charlie") # ✅ No error!
print(google.employees) # ['Alice', 'Bob', 'Charlie']

# The list itself is mutable, even though the field reference is frozen

Solutions for Deep Immutability​

Option 1: Use Immutable Types

from typing import Tuple

@dataclass(frozen=True)
class ImmutableCompany:
name: str
employees: Tuple[str, ...] # ✅ Tuple is immutable

company = ImmutableCompany("Google", ("Alice", "Bob"))
# company.employees.append("Charlie") # ❌ AttributeError: 'tuple' object has no attribute 'append'

Option 2: Convert to Immutable in __post_init__

from dataclasses import dataclass, field
from typing import List

@dataclass(frozen=True)
class SafeCompany:
name: str
employees: List[str] = field(default_factory=list)

def __post_init__(self):
# Convert list to tuple after initialization
object.__setattr__(self, 'employees', tuple(self.employees))

company = SafeCompany("Google", ["Alice", "Bob"])
print(company.employees) # ('Alice', 'Bob')
# company.employees.append("Charlie") # ❌ AttributeError

Option 3: Use functools.cached_property for Computed Mutable Data

from dataclasses import dataclass
from functools import cached_property
from typing import Dict

@dataclass(frozen=True)
class Analytics:
user_ids: Tuple[int, ...]

@cached_property
def user_id_to_index(self) -> Dict[int, int]:
"""Compute once, cache forever"""
return {user_id: idx for idx, user_id in enumerate(self.user_ids)}

analytics = Analytics((101, 102, 103))
print(analytics.user_id_to_index) # {101: 0, 102: 1, 103: 2}

Working with Immutable Objects​

The replace() Function Pattern​

from dataclasses import dataclass, replace

@dataclass(frozen=True)
class UserProfile:
username: str
email: str
is_active: bool = True
permissions: Tuple[str, ...] = ()

def deactivate(self) -> 'UserProfile':
"""Functional style: return new object with changes"""
return replace(self, is_active=False)

def add_permission(self, permission: str) -> 'UserProfile':
return replace(
self,
permissions=self.permissions + (permission,)
)

# Chain updates functionally
user = UserProfile("alice", "alice@example.com")
user = user.add_permission("read").add_permission("write")
user = user.deactivate()

print(user) # UserProfile(username='alice', email='alice@example.com', is_active=False, permissions=('read', 'write'))

Bulk Updates with Dictionary Unpacking​

@dataclass(frozen=True)
class Settings:
theme: str
language: str
notifications: bool
timeout: int

settings = Settings("dark", "en", True, 30)

# Update multiple fields at once
new_settings = replace(
settings,
**{"theme": "light", "notifications": False} # Can use dict unpacking
)

# Or with variable updates
updates = {}
if user_prefers_light_mode:
updates["theme"] = "light"
if user_is_busy:
updates["notifications"] = False

updated_settings = replace(settings, **updates)

Versioned Data with Inheritance​

from dataclasses import dataclass
from typing import Optional

@dataclass(frozen=True)
class UserV1:
id: int
name: str

@dataclass(frozen=True)
class UserV2(UserV1):
email: Optional[str] = None
phone: Optional[str] = None

# Backward compatible
v1_user = UserV1(1, "Alice")
v2_user = UserV2(1, "Alice", email="alice@example.com")

# Type checking works
def process_user(user: UserV1):
print(f"Processing: {user.name}")

process_user(v1_user) # ✅
process_user(v2_user) # ✅ (inheritance)

Advanced Patterns & Alternatives​

Pattern 1: Builder Pattern for Complex Objects​

from dataclasses import dataclass, field
from typing import List, Optional

@dataclass(frozen=True)
class Query:
table: str
columns: Tuple[str, ...]
filters: Tuple['Filter', ...]
limit: Optional[int] = None

@dataclass
class Builder:
"""Mutable builder for immutable Query"""
table: str = ""
columns: List[str] = field(default_factory=list)
filters: List['Filter'] = field(default_factory=list)
limit: Optional[int] = None

def build(self) -> 'Query':
return Query(
table=self.table,
columns=tuple(self.columns),
filters=tuple(self.filters),
limit=self.limit
)

@dataclass(frozen=True)
class Filter:
column: str
operator: str
value: any

# Usage
builder = Query.Builder()
builder.table = "users"
builder.columns = ["id", "name", "email"]
builder.filters.append(Query.Filter("age", ">", 18))
builder.limit = 100

query = builder.build() # ✅ Immutable Query object

Pattern 2: Registry Pattern with Frozen Classes​

from dataclasses import dataclass
from typing import Dict, ClassVar

@dataclass(frozen=True)
class ErrorCode:
code: int
message: str

# Registry of all error codes
_registry: ClassVar[Dict[int, 'ErrorCode']] = {}

def __post_init__(self):
# Register the instance
ErrorCode._registry[self.code] = self

@classmethod
def get(cls, code: int) -> 'ErrorCode':
return cls._registry[code]

# Define error codes (these become singletons)
NOT_FOUND = ErrorCode(404, "Resource not found")
UNAUTHORIZED = ErrorCode(401, "Authentication required")
FORBIDDEN = ErrorCode(403, "Insufficient permissions")

# Safe to use anywhere
print(ErrorCode.get(404)) # ErrorCode(code=404, message='Resource not found')

Alternative 1: collections.namedtuple​

from collections import namedtuple
from typing import NamedTuple

# Basic immutable structure
Point = namedtuple('Point', ['x', 'y'])
p = Point(1, 2)
# p.x = 3 # ❌ AttributeError: can't set attribute

# With type hints (Python 3.6+)
class PointNT(NamedTuple):
x: float
y: float
z: float = 0.0

# Limitations: Less flexible than dataclasses, no easy field defaults

Alternative 2: attrs Library​

import attr

@attr.s(frozen=True, slots=True)
class AttrsPoint:
x: float = attr.ib()
y: float = attr.ib()
color: str = attr.ib(default="blue")

# Additional features like validators, converters
@attr.s(frozen=True)
class ValidatedPoint:
x: float = attr.ib(validator=attr.validators.instance_of(float))
y: float = attr.ib(validator=attr.validators.instance_of(float))

Performance Considerations​

Memory and Speed Implications​

import timeit
from dataclasses import dataclass, make_dataclass
from pympler import asizeof

@dataclass(frozen=True)
class FrozenUser:
id: int
name: str
email: str

@dataclass
class MutableUser:
id: int
name: str
email: str

# Memory usage
frozen = FrozenUser(1, "Alice", "alice@example.com")
mutable = MutableUser(1, "Alice", "alice@example.com")

print(f"Frozen size: {asizeof.asizeof(frozen)} bytes")
print(f"Mutable size: {asizeof.asizeof(mutable)} bytes")
# Typically very similar - frozen might be slightly larger due to __hash__

# Creation speed
setup = "from __main__ import FrozenUser, MutableUser"
stmt_frozen = "FrozenUser(1, 'Alice', 'alice@example.com')"
stmt_mutable = "MutableUser(1, 'Alice', 'alice@example.com')"

time_frozen = timeit.timeit(stmt_frozen, setup=setup, number=1000000)
time_mutable = timeit.timeit(stmt_mutable, setup=setup, number=1000000)

print(f"Frozen creation: {time_frozen:.3f}s")
print(f"Mutable creation: {time_mutable:.3f}s")
# Frozen is slightly slower due to additional checks

When Performance Matters​

# For high-performance scenarios, consider __slots__
@dataclass(frozen=True)
class OptimizedPoint:
__slots__ = ('x', 'y', 'z') # Reduces memory overhead
x: float
y: float
z: float = 0.0

# Or use tuple directly for maximum performance
from typing import NamedTuple

class PointNT(NamedTuple):
x: float
y: float
z: float = 0.0

# Benchmark: NamedTuple is fastest for creation and memory

Best Practices​

1. Default to Frozen​

# Good: Start with frozen, relax only if needed
@dataclass(frozen=True)
class Config:
api_key: str
timeout: int = 30
retries: int = 3

# Only make mutable if you have a proven need
@dataclass # No frozen - use sparingly!
class Session:
user_id: int
token: str
expiry: float

2. Use Immutable Field Types​

from typing import Tuple, FrozenSet

@dataclass(frozen=True)
class ImmutableData:
# Good: Immutable types
ids: Tuple[int, ...]
tags: FrozenSet[str]
name: str
count: int

# Avoid (unless converted in __post_init__):
# items: List[str] # ❌
# config: Dict[str, Any] # ❌

3. Implement Domain Logic as Methods​

@dataclass(frozen=True)
class Money:
amount: float
currency: str

def add(self, other: 'Money') -> 'Money':
if self.currency != other.currency:
raise ValueError("Currencies must match")
return replace(self, amount=self.amount + other.amount)

def convert(self, rate: float, target_currency: str) -> 'Money':
return Money(amount=self.amount * rate, currency=target_currency)

usd = Money(100, "USD")
eur = usd.convert(0.85, "EUR")
total = usd.add(Money(50, "USD"))

4. Combine with kw_only for Clarity​

@dataclass(frozen=True, kw_only=True)
class ServerConfig:
host: str
port: int
ssl: bool = True
timeout: int = 30

# Forces explicit parameter names
config = ServerConfig(host="example.com", port=443)
# Not: ServerConfig("example.com", 443) # ❌

5. Document Frozen Assumptions​

@dataclass(frozen=True)
class APIResponse:
"""Immutable response from API.

This class is frozen to ensure:
1. Thread safety when sharing between components
2. Hashability for caching
3. Predictable behavior in functional pipelines

Use `dataclasses.replace()` to create modified copies.
"""
status: int
data: dict
headers: Tuple[Tuple[str, str], ...]

Common Pitfalls and Solutions​

Pitfall 1: Inadvertent Mutation of Nested Objects​

# ❌ Problem
@dataclass(frozen=True)
class Graph:
nodes: dict

# ✅ Solution
@dataclass(frozen=True)
class Graph:
nodes: frozendict # Requires 'frozendict' package
# OR
nodes: Mapping[str, Any] # Read-only interface

# ✅ Better Solution
@dataclass(frozen=True)
class Graph:
_nodes: dict = field(default_factory=dict, repr=False)

@property
def nodes(self) -> Mapping[str, Any]:
return MappingProxyType(self._nodes) # Read-only view

Pitfall 2: Circular References​

from typing import Optional

@dataclass(frozen=True)
class TreeNode:
value: any
left: Optional['TreeNode'] = None
right: Optional['TreeNode'] = None

# This works! Frozen doesn't prevent object graph formation
# It only prevents reassignment of the fields themselves

Pitfall 3: Serialization Issues​

import json
from dataclasses import dataclass, asdict

@dataclass(frozen=True)
class Data:
x: int
y: int

d = Data(1, 2)

# Standard serialization works
json_str = json.dumps(asdict(d))

# For custom serialization, implement __json__ or use a mixin
import json
from abc import ABC

class JSONSerializable(ABC):
def to_json(self) -> str:
return json.dumps(asdict(self))

@dataclass(frozen=True)
class JsonData(JSONSerializable):
x: int
y: int

Conclusion​

@dataclass(frozen=True) is a powerful tool that brings functional programming benefits to Python's object-oriented world. It provides:

  1. Safety through immutability
  2. Convenience through automatic method generation
  3. Expressiveness through clean, declarative syntax
  4. Performance through hashability and thread safety

While the "shallow freezing" limitation requires attention, following the patterns and best practices outlined above will help you create robust, maintainable, and bug-resistant data models in Python.

When to use it: Configuration objects, value objects, messages/events, cache keys, shared state in concurrent systems, and anywhere you want to enforce the principle of "no surprises."

When to avoid it: When you genuinely need mutable state, when working with large object graphs that change frequently, or when the performance overhead of copying is prohibitive.

Using print for Python Debugging? You're Outdated!

· 4 min read

In Python development practice, debugging is an indispensable part of the process. If you rely on print() statements to trace the execution flow of your program, you might encounter persistent exceptions that are difficult to pinpoint even after multiple code reviews. This is often due to the limitations of this debugging method as the terminal output grows, making it hard to correlate outputs with their corresponding function calls. This article introduces the IceCream library, a specialized debugging tool that significantly enhances debugging efficiency, making the entire process more systematic and standardized.

print() is the most basic output function in Python and is the preferred debugging tool for most developers. However, when dealing with complex function calls and data structures, this method often leads to cluttered output, reducing debugging efficiency. The ic() function from the IceCream library is optimized for debugging scenarios, providing more practical features.

Basic Debugging Example - Using print​

def add(x, y):
return x + y

# Using print() for function debugging
print(add(10, 20)) # Output: 30
print(add(30, 40)) # Output: 70

The main issue with this traditional method is that when there are many output results, it becomes difficult to intuitively associate the output values with their corresponding function calls. Additional manual explanations are required.

Using ic for Debugging​

from icecream import ic

# Using ic() for function debugging
ic(add(10, 20))
ic(add(30, 40))

Output:

ic| add(10, 20): 30
ic| add(30, 40): 70

By using the ic() function, each output clearly displays the complete information of the function call, including the function name, parameter values, and return results. This output format is particularly suitable for debugging complex sequences of function calls, allowing for quick problem identification.

Core Advantages of the ic Function​

  1. Detailed Execution Information Tracking

    The ic() function not only displays the execution results but also fully records the operation process, eliminating the need for manually writing debugging information and improving debugging efficiency.

    def multiply(a, b):
    return a * b

    ic(multiply(5, 5))

    Output:

    ic| multiply(5, 5): 25
  2. Integration of Debugging and Assignment Operations

    One notable feature of the ic() function is its support for simultaneous debugging and variable assignment, which is not possible with traditional print() functions:

    # print() method
    result = print(multiply(4, 6)) # Output: 24
    print(result) # Output: None

    # ic() method
    result = ic(multiply(4, 6)) # Output: ic| multiply(4, 6): 24
    print(result) # Output: 24

    Using the ic() function, you can not only view debugging information but also correctly obtain and store the return value, which is particularly useful during debugging.

  3. Visualization of Data Structure Access

    When dealing with data structures like dictionaries, the ic() function provides clearer access information:

    data = {'a': 1, 'b': 2, 'c': 3}

    # Using ic() to track data access
    ic(data['a'])

    Output:

    ic| data['a']: 1

    The output clearly shows the access path and result, helping to understand the data operation process.

  4. Optimization of Complex Data Structure Display

    When dealing with nested dictionaries or JSON-like complex data structures, the ic() function provides better readability through structured formatting:

    complex_data = {
    "name": "John",
    "age": 30,
    "languages": ["Python", "JavaScript"]
    }

    ic(complex_data)

    The output uses structured formatting with color differentiation, significantly enhancing the readability of complex data structures, facilitating quick location and analysis of data.

Advanced Features of the IceCream Library​

In addition to basic debugging functions, the IceCream library offers a series of advanced features that can be customized according to specific needs:

  1. Dynamic Control of Debugging Output

    During development, you can dynamically control the output of debugging information as needed:

    ic.disable() # Pause debugging output
    ic(multiply(3, 3)) # No output here

    ic.enable() # Resume debugging output
    ic(multiply(3, 3)) # Output: ic| multiply(3, 3): 9
  2. Custom Configuration of Output Format

    IceCream supports custom output formats, allowing you to adjust the output method according to project requirements:

    def log_to_file(text):
    with open("debug.log", "a") as f:
    f.write(text + "\n")

    ic.configureOutput(prefix="DEBUG| ", outputFunction=log_to_file)

    ic(multiply(7, 7))

    This configuration can redirect debugging information to a log file and add a custom prefix, facilitating subsequent log analysis.

Conclusion​

Although the print() function is widely used as a basic debugging tool in Python, it has obvious limitations in complex development scenarios. The IceCream library effectively addresses the shortcomings of traditional debugging methods by providing more professional debugging tools. Its rich features, flexible configuration options, and clear output format can significantly enhance the debugging efficiency of Python programs. In practical development, using the ic() function properly not only helps developers locate and solve problems faster but also improves code maintainability.

Deep Dive into Finding Numbers in Strings with Python

· 6 min read

Introduction​

Handling strings that contain a mix of numbers and text is a common task in everyday programming. For example, you might need to extract numbers from complex text or identify whether certain strings contain numbers. In Python, there are various ways to tackle this problem, depending on your specific needs. This article will detail how to find numbers in strings using Python, providing concrete example code to help developers quickly master these techniques.


Why Find Numbers in Strings?​

  • Data Processing and Cleaning: Extract useful numeric information from web crawlers or log files, such as order numbers, timestamps, amounts, etc.
  • Input Validation: Detect whether user input strings contain numbers to ensure the validity of the input content.
  • Text Analysis: Extract numbers from text data for further data analysis or statistics.

Using String's isnumeric() Method​

Python's string type (str) provides several methods to determine character types, one of which is isnumeric(). The isnumeric() method checks whether a string contains only numeric characters. If all characters in the string are numeric, the method returns True; otherwise, it returns False.

string = "12345"
if string.isnumeric():
print(f"'{string}' contains only numbers")
else:
print(f"'{string}' contains more than just numbers")

In this example, isnumeric() checks if the string "12345" consists entirely of numeric characters. Although this method can determine if a string is entirely numeric, it cannot extract numbers from mixed strings.


Using Regular Expressions to Find Numbers​

Regular expressions (regex) are a powerful tool for text processing, ideal for finding and extracting numbers from strings. Python's re module provides comprehensive regex support, making it easy to match numbers in strings.

Example: Using Regular Expressions​

import re

# Define a string containing numbers
text = "Order number is 12345, amount is 678.90 dollar"

# Use regex to find all numbers
numbers = re.findall(r'\d+', text)

print(f"Numbers found in the string: {numbers}")

In this example, re.findall() uses the regex \d+, which matches one or more consecutive digits. re.findall() returns a list containing all matched numbers in the string. The output is: ['12345', '678'].


Finding Numbers with Decimal Points​

If you need to find numbers with decimal points in a string, you can use the following regex:

import re

text = "Order number is 12345, amount is 678.90 dollar"

# Find integers and decimals
numbers = re.findall(r'\d+\.?\d*', text)

print(f"Numbers found in the string: {numbers}")

In this regex, \d+ matches one or more digits, \. matches the decimal point, and \d* matches zero or more digits following the decimal point. The result is: ['12345', '678.90'].


Using List Comprehension to Extract Numbers​

For simple scenarios, you can use Python's list comprehension to quickly extract numbers from a string. By checking each character in the string to see if it is a digit, you can extract them.

Example: Using List Comprehension​

text = "Order number 12345, amount 678.90 dollar"

# Extract digit characters
digits = [char for char in text if char.isdigit()]

print(f"Extracted digit characters: {digits}")

In this example, isdigit() is used to check each character in the string, resulting in a list of all digit characters: ['1', '2', '3', '4', '5', '6', '7', '8', '9', '0']. If you need to combine the extracted digits into a complete numeric string, you can use the join() method:

# Combine extracted digit characters into a complete numeric string
number_str = ''.join(digits)
print(f"Complete numeric string: {number_str}")

The output is: "1234567890".


Filtering Numbers with filter() Function​

Python's built-in filter() function can be used to filter out non-numeric characters from a string, thereby extracting numeric characters. The filter() function works similarly to list comprehension but is more functional in style.

Example: Using filter() Function​

text = "Order number 12345, amount 678.90 dollar"

# Use the filter function to extract numbers
numbers = ''.join(filter(str.isdigit, text))

print(f"Extracted numbers: {numbers}")

In this example, the filter() function checks each character in the string, keeping only those that satisfy isdigit(), and then joins the extracted numeric characters into a complete string.


Splitting Strings to Extract Numbers​

In some scenarios, you can split the string first and then process each segment to extract numbers. The split() method can split a string into multiple parts based on a specified delimiter, suitable for handling structured text.

Example: Using split() Method​

import re

text = "Order number: 12345, amount: 678.90"

# Split the string by spaces or punctuation
parts = re.split(r'[,:: ]+', text)

# Extract numeric parts
numbers = [part for part in parts if part.replace('.', '', 1).isdigit()]

print(f"Extracted numbers: {numbers}")

In this example, re.split() is used to split the string, and then list comprehension extracts all numeric parts. replace('.', '', 1) ensures that decimal points are handled correctly.


Using ast.literal_eval to Parse Numbers​

In some applications, you may need to extract numeric values from complex strings and parse them into numeric types (integers or floats). Python's ast.literal_eval() method can safely parse a string into a Python data type.

Example: Using ast.literal_eval​

import ast
import re

text = "Amount: 678.90"

# Extract the numeric part of the string
number_str = re.search(r'\d+\.?\d*', text).group()

# Parse the string into a number
number = ast.literal_eval(number_str)

print(f"Parsed number: {number}")

In this example, the regex extracts the numeric part, and ast.literal_eval() converts it into the actual numeric type (float). This method is useful for extracting and converting numbers from text.


Conclusion​

Finding and extracting numbers from strings is a common task in Python programming, whether for data cleaning, user input validation, or text analysis. This article has introduced various methods to find numbers in Python, including using string methods, regular expressions, list comprehension, the filter() function, and ast.literal_eval(). Each method has its strengths and can be chosen based on different application scenarios. With these tools, developers can quickly and efficiently extract and process numeric data from strings, enhancing code flexibility and processing capabilities.

Restman

· 2 min read
adfad
111

Restman is a command-line tool written in Python, designed to combine the simplicity of cURL with the powerful features of Postman. It aims to relieve users from the complexities of manually building network connections with URLs or endpoints, allowing them to focus solely on the core aspects of HTTP requests and responses. With Restman, users can quickly and efficiently test APIs, handle different HTTP methods, and parse response data without worrying about the underlying connection mechanics.

Github repo​

WhereQ Commandoes --- Restman

Free public APIs for Testing​

WebsiteURLDescription
Dummy API Exampledummy.restapiexample.comA simple API for testing dummy data.
ReqResreqres.inA hosted REST-API to simulate CRUD operations for testing.
Cat Factscatfact.ninjaProvides random cat facts.
Dog CEO APIdog.ceoRandom dog image generator.
JSONPlaceholderjsonplaceholder.typicode.comFake API for testing and prototyping.
RestCountriesrestcountries.comAPI for getting country-related information.
PokeAPIpokeapi.coFree API for Pokémon data.
CoinDesk APIcoindesk.comProvides Bitcoin price index and cryptocurrency news.
Agifyapi.agify.ioPredicts the age of a person based on their name.
Genderizeapi.genderize.ioPredicts the gender of a person based on their name.
Nationalizeapi.nationalize.ioPredicts the nationality based on a name.
Data USAdatausa.ioProvides data on the U.S. population.
IPifyapi.ipify.orgSimple API to get your public IP address.
IPInfoipinfo.ioProvides detailed geolocation information of an IP address.
Official Joke APIofficial-joke-api.appspot.comFetches a random joke.
Random User APIrandomuser.meGenerates random user data.
University Search APIuniversities.hipolabs.comSearch for universities worldwide.
Zippopotamzippopotam.usGet location data by U.S. ZIP code.