跳到主要内容

4 篇博文 含有标签「ai」

查看所有标签

Full Fine-Tuning vs LoRA

· 阅读需 21 分钟

Table of Contents​

  1. Introduction
  2. What is Fine-Tuning?
  3. Full Fine-Tuning Explained
  4. LoRA (Low-Rank Adaptation) Explained
  5. Technical Comparison
  6. Real-World Use Cases
  7. When to Use Which Method
  8. Hands-On Practice Guide
  9. Advanced Topics
  10. Decision Framework

Introduction​

In the era of Large Language Models (LLMs), fine-tuning has become the key to adapting pre-trained models to specific tasks. Two main approaches dominate the landscape:

  • Full Fine-Tuning: Training all model parameters
  • LoRA (Low-Rank Adaptation): Training only a small subset of parameters

What is Fine-Tuning?​

Simple Analogy​

Think of a pre-trained model as a medical doctor with general knowledge. Fine-tuning is like:

  • Full Fine-Tuning: Sending the doctor back to medical school to relearn everything with a focus on a specialty
  • LoRA: Giving the doctor a specialized handbook they can reference without forgetting their general knowledge

Technical Definition​

Fine-tuning takes a pre-trained model (trained on massive general data) and continues training it on domain-specific data to improve performance on particular tasks.

Pre-trained Model (General Knowledge)
↓
Fine-Tuning (Specialized Training)
↓
Task-Specific Model (Expert Knowledge)

Full Fine-Tuning Explained​

What is Full Fine-Tuning?​

Full fine-tuning means updating ALL parameters in the model during training.

Model Architecture (Example: 7B parameters)
┌──────────────────────────────────────┐
│ Layer 1: [All parameters trainable] │ ← Updated
│ Layer 2: [All parameters trainable] │ ← Updated
│ Layer 3: [All parameters trainable] │ ← Updated
│ ... │ ← Updated
│ Layer N: [All parameters trainable] │ ← Updated
└──────────────────────────────────────┘
Total Trainable: 7,000,000,000 params

How It Works​

  1. Load pre-trained model (e.g., GPT-3, LLaMA, ChatGLM)
  2. Prepare task-specific dataset (e.g., medical Q&A, legal documents)
  3. Train ALL parameters using your dataset
  4. Save the entire model (full 7B+ parameters)

Advantages ✅​

  1. Maximum Performance: Can achieve the best possible results
  2. Deep Customization: All parameters adapt to your specific task
  3. Flexible: Works for any task, especially those very different from pre-training

Disadvantages ❌​

  1. Extremely High Cost:

    • GPT-3 175B full fine-tuning requires >2TB GPU memory
    • Needs high-end GPU clusters (multiple A100s)
    • Training can cost $20,000+ per run
  2. Long Training Time: Days or weeks to complete

  3. Storage Overhead:

    • Each task needs a complete model copy
    • 7B model = ~14GB storage per task
    • 10 tasks = 140GB storage needed
  4. Catastrophic Forgetting:

    • Model may lose general knowledge
    • Reduced generalization ability

Real-World Example​

Google's Multilingual Search Optimization

  • Used full fine-tuning on BERT-Large for multilingual search
  • Result: 25% improvement in low-resource language recall
  • Cost: Requires massive GPU infrastructure
  • Benefit: State-of-the-art performance across 100+ languages

LoRA (Low-Rank Adaptation) Explained​

What is LoRA?​

LoRA is a parameter-efficient fine-tuning method proposed by Microsoft in 2021. Instead of updating all parameters, LoRA:

  • Freezes the original model weights
  • Adds small "adapter" matrices to specific layers
  • Trains only these tiny adapters (typically 0.1%-1% of total parameters)

Visual Explanation​

Original Transformer Layer
┌────────────────────────────────────────┐
│ Pre-trained Weight Matrix W │
│ [Frozen - Not Updated] │
│ + │
│ Low-Rank Adapter: A × B │
│ [Trainable - Only These Updated] │
└────────────────────────────────────────┘

Output = W·x + A·B·x
↑ ↑
Frozen Trained (tiny!)

Mathematical Foundation​

For a pre-trained weight matrix W (dimension d × k):

Full Fine-Tuning:
Updates: ΔW (d × k parameters)
Total trainable: d × k

LoRA:
Decomposes: ΔW ≈ A × B
Where: A is (d × r), B is (r × k)
Total trainable: d×r + r×k

Example (d=4096, k=4096, r=8):
Full: 4096 × 4096 = 16,777,216 params
LoRA: 4096×8 + 8×4096 = 65,536 params
Reduction: 99.6%! 🎉

How LoRA Works (Step-by-Step)​

  1. Freeze Original Model

    for param in pretrained_model.parameters():
    param.requires_grad = False # Don't update these
  2. Inject Low-Rank Matrices

    • Insert matrices A and B into attention layers
    • Initialize: A with random values, B with zeros
    • Typically apply to Query, Value projection matrices
  3. Train Only Adapters

    for param in lora_adapters.parameters():
    param.requires_grad = True # Only update these
  4. Inference

    output = original_weight @ x + (A @ B) @ x

Advantages ✅​

  1. Extreme Parameter Efficiency:

    • Reduces trainable parameters by 90-99%
    • 7B model: Train only 7-70M parameters instead of 7B
  2. Low Cost, Fast Deployment:

    • Consumer GPU (RTX 4090, RTX 3090) can fine-tune large models
    • Training time: Hours instead of days
    • Cost: **50−200∗∗insteadof50-200** instead of 20,000+
  3. Multi-Task Flexibility:

    • Save only adapter weights (~10-100MB per task)
    • Dynamically load different adapters for different tasks
    • One base model + multiple adapters = multiple specialized models
  4. Less Catastrophic Forgetting:

    • Original weights unchanged
    • Retains general knowledge better

Disadvantages ❌​

  1. Rank Parameter Tuning Required:

    • Too small r (rank) → underfitting
    • Too large r → loses efficiency
    • Needs experimentation
  2. "Intruder Dimensions" Problem (Recent Discovery):

    • LoRA introduces high-ranking singular vectors
    • Reduces out-of-distribution (OOD) generalization by 5-8%
    • Less robust in continual learning scenarios
  3. Slightly Lower Peak Performance:

    • May not reach absolute best performance
    • Gap is usually small (<1-3%) for most tasks

Real-World Examples​

1. Medical Document Summarization

  • Hospital used LoRA to fine-tune GPT-3 for medical report generation
  • Result: High-quality summaries, doctor efficiency increased 40%
  • Cost: Single RTX A6000 GPU, trained in 6 hours
  • Storage: Base model (14GB) + Adapter (50MB)

2. E-commerce Customer Service

  • Online retailer fine-tuned ChatGLM for customer support
  • Result: Response speed improved 3x, satisfaction up 25%
  • Deployment: 5 different product category adapters
  • Total storage: One model + 5 adapters = 14.3GB (vs 70GB for 5 full models)

3. Financial News Generation

  • News agency fine-tuned GPT-3 for real-time financial reporting
  • Result: News generation latency reduced to seconds, 40% cost reduction
  • Adaptation time: 4 hours on 2× RTX 4090

Technical Comparison​

Architecture Comparison​

FULL FINE-TUNING
════════════════════════════════════════
Input → [Layer 1 ✏️] → [Layer 2 ✏️] → [Layer 3 ✏️] → Output
All Updated All Updated All Updated

LoRA
════════════════════════════════════════
Input → [Layer 1 🔒] → [Layer 2 🔒] → [Layer 3 🔒] → Output
+ Adapter✏️ + Adapter✏️ + Adapter✏️

🔒 = Frozen (unchanged)
✏️ = Trainable (updated during training)

Detailed Comparison Table​

AspectFull Fine-TuningLoRA
Trainable Parameters100% (e.g., 7B params)0.1-1% (e.g., 7-70M params)
GPU MemoryVery High (>80GB for 7B)Low (24-48GB for 7B)
Training TimeDays to weeksHours to 1-2 days
Cost per Training10,000−10,000 - 100,000+50−50 - 500
Storage per TaskFull model (~14GB for 7B)Adapter only (~10-100MB)
PerformanceHighest (100% baseline)Very High (95-99% of full)
Multi-Task DeploymentDifficult (one model per task)Easy (one model + N adapters)
Catastrophic ForgettingHigh riskLow risk
Hardware RequirementsHigh-end GPU cluster (A100×8)Consumer GPU (RTX 3090/4090)
Generalization (OOD)BestGood (5-8% lower)
Best ForMaximum performance, unlimited budgetFast iteration, limited resources

Memory Comparison (7B Model Example)​

Full Fine-Tuning (7B parameters):
┌─────────────────────────────────┐
│ Model: 14GB │
│ Optimizer States: 42GB │
│ Gradients: 14GB │
│ Activations: 20GB │
│ ─────────────────────────────── │
│ Total: ~90GB │
└─────────────────────────────────┘
Required: 4× A100 (40GB each)

LoRA (7B base + 10M adapter):
┌─────────────────────────────────┐
│ Base Model: 14GB (frozen) │
│ Adapter: 20MB │
│ Optimizer States: 60MB │
│ Gradients: 20MB │
│ Activations: 12GB │
│ ─────────────────────────────── │
│ Total: ~27GB │
└─────────────────────────────────┘
Required: 1× RTX 3090/4090 (24GB)

Real-World Use Cases​

Use Case 1: Healthcare - Clinical Note Generation​

Scenario: Hospital needs AI to generate structured clinical notes from doctor-patient conversations.

Challenge:

  • Medical terminology requires domain adaptation
  • HIPAA compliance = can't use cloud APIs
  • Limited budget and GPU resources

Solution: LoRA Fine-Tuning

# Approach
Base Model: LLaMA-2-7B
Dataset: 10,000 anonymized clinical conversations
Training: LoRA (r=16, α=32)
Hardware: 1× RTX A6000 (48GB)
Time: 8 hours
Cost: ~$100

# Results
- Accuracy: 94% (vs 96% with full fine-tuning)
- Deployment: One base model + adapter (14.05GB total)
- Multi-specialty: 5 adapters for different departments (14.3GB total)

Why LoRA Won:

  • ✅ Budget-friendly
  • ✅ Fast iteration for different specialties
  • ✅ Meets accuracy requirements
  • ✅ Easy deployment on-premise

Scenario: Law firm needs to analyze merger & acquisition contracts for risk factors.

Challenge:

  • Complex legal language
  • High accuracy requirement (mistakes = legal liability)
  • Contracts vary significantly by jurisdiction

Solution: Full Fine-Tuning

# Approach
Base Model: GPT-3.5-turbo-instruct
Dataset: 50,000 labeled M&A contracts
Training: Full fine-tuning all parameters
Hardware: 8× A100 GPUs (cloud)
Time: 4 days
Cost: ~$25,000

# Results
- Accuracy: 98.5% (vs 95% with LoRA r=64)
- Risk detection: 99.2% recall (critical for legal)
- False positive rate: 1.8% (vs 4.2% with LoRA)

Why Full Fine-Tuning Won:

  • ✅ Maximum accuracy needed (legal liability)
  • ✅ Budget available (law firm can afford)
  • ✅ Single specialized task
  • ✅ 3% accuracy improvement = millions in risk mitigation

Use Case 3: E-Commerce - Multi-Category Product Descriptions​

Scenario: Online marketplace needs to generate product descriptions for 10 different categories (electronics, fashion, home, etc.).

Challenge:

  • 10 different writing styles needed
  • Rapid product launches (new categories added quarterly)
  • Budget-conscious startup

Solution: LoRA with Multi-Adapter Architecture

# Approach
Base Model: Qwen-7B
Training: 10 separate LoRA adapters (r=8)
- Electronics adapter
- Fashion adapter
- Home & Garden adapter
- Sports adapter
- ... (6 more)
Hardware: 1× RTX 4090
Time per adapter: 3 hours
Total cost: ~$300

# Deployment
Storage:
Base model: 14GB
10 adapters: 10 × 40MB = 400MB
Total: 14.4GB (vs 140GB for 10 full models)

# Runtime
Load base model once, swap adapters dynamically:
electronics_model = base + electronics_adapter
fashion_model = base + fashion_adapter

Why LoRA Won:

  • ✅ 10× storage savings
  • ✅ Can train new category adapter in hours
  • ✅ Single GPU deployment
  • ✅ Easy A/B testing different adapters

Use Case 4: Multilingual Customer Support​

Scenario: Global SaaS company needs chatbots for 20 languages.

Challenge:

  • 20 language-specific models needed
  • Need to update all models when features change
  • Some languages low-resource (limited training data)

LoRA Approach (Recommended):

Base Model: Multilingual LLaMA-2-13B
Training: 20 language adapters (r=16)

Storage: 26GB base + (20 × 80MB) = 27.6GB
vs Full Fine-Tuning: 20 × 26GB = 520GB

Benefit: Update base model → all languages benefit

When to Use Which Method​

Choose Full Fine-Tuning When:​

  1. Maximum Performance is Critical ✅

    • Legal, medical, financial applications where errors are costly
    • Need that extra 2-5% accuracy
  2. Large Dataset Available ✅

    • Have >100,000 high-quality labeled examples
    • Data justifies training all parameters
  3. Unlimited Budget & Resources ✅

    • Access to GPU clusters (8+ A100s)
    • Can afford $10,000-100,000 training costs
  4. Single Specialized Task ✅

    • Only need one model for one task
    • No plan for multi-task deployment
  5. Task Very Different from Pre-training ✅

    • Specialized domains (medical, legal, scientific)
    • Low-resource languages

Example Decision:

Use Full Fine-Tuning if:
(Accuracy_requirement > 97%) AND
(Budget > $10,000) AND
(Single_task OR Task_very_different)

Choose LoRA When:​

  1. Limited Resources ✅

    • Budget < $5,000
    • Only have consumer GPUs (RTX 3090, 4090)
  2. Need Fast Iteration ✅

    • Startup environment
    • Frequent model updates
    • Rapid experimentation
  3. Multi-Task Deployment ✅

    • Need models for 5+ different tasks/domains
    • Want to switch between tasks dynamically
  4. Storage Constraints ✅

    • Limited disk space
    • Edge device deployment
  5. Continuous Learning ✅

    • Need to add new capabilities over time
    • Can't afford catastrophic forgetting

Example Decision:

Use LoRA if:
(Budget < $5,000) OR
(GPU_memory < 80GB) OR
(Num_tasks > 3) OR
(Need_fast_iteration)

Hands-On Practice Guide​

Prerequisites​

# Install required libraries
pip install torch transformers peft datasets accelerate bitsandbytes

# Check GPU
nvidia-smi

Practice 1: Full Fine-Tuning (Small Model)​

Task: Fine-tune GPT-2 (small) for custom text generation

from transformers import GPT2LMHeadModel, GPT2Tokenizer, Trainer, TrainingArguments
from datasets import load_dataset

# 1. Load pre-trained model
model = GPT2LMHeadModel.from_pretrained("gpt2")
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
tokenizer.pad_token = tokenizer.eos_token

# 2. Prepare dataset
dataset = load_dataset("wikitext", "wikitext-2-raw-v1")

def tokenize_function(examples):
return tokenizer(examples["text"], truncation=True,
max_length=512, padding="max_length")

tokenized_datasets = dataset.map(tokenize_function, batched=True)

# 3. Training configuration (FULL FINE-TUNING)
training_args = TrainingArguments(
output_dir="./gpt2-finetuned-full",
num_train_epochs=3,
per_device_train_batch_size=4,
save_steps=500,
save_total_limit=2,
learning_rate=2e-5,
# All parameters trainable (default)
)

# 4. Train
trainer = Trainer(
model=model,
args=training_args,
train_dataset=tokenized_datasets["train"],
)

trainer.train()

# 5. Save model
model.save_pretrained("./gpt2-finetuned-full")
tokenizer.save_pretrained("./gpt2-finetuned-full")

print(f"Model size: {sum(p.numel() for p in model.parameters())} parameters")
# Output: ~124M parameters all trained

Resource Requirements:

  • GPU: GTX 1080 Ti or better (11GB+)
  • Time: ~2 hours
  • Storage: ~500MB

Practice 2: LoRA Fine-Tuning (Large Model)​

Task: Fine-tune LLaMA-2-7B for instruction following using LoRA

from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from datasets import load_dataset
from trl import SFTTrainer

# 1. Load base model (with 4-bit quantization for efficiency)
model_name = "meta-llama/Llama-2-7b-hf"
model = AutoModelForCausalLM.from_pretrained(
model_name,
load_in_4bit=True, # Quantization to reduce memory
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token

# 2. Configure LoRA
lora_config = LoraConfig(
r=16, # Rank (try 8, 16, 32, 64)
lora_alpha=32, # Scaling factor
target_modules=["q_proj", "v_proj"], # Apply to attention
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM"
)

# 3. Prepare model for LoRA training
model = prepare_model_for_kbit_training(model)
model = get_peft_model(model, lora_config)

# Print trainable parameters
model.print_trainable_parameters()
# Output: trainable params: 4,194,304 || all params: 6,738,415,616 || trainable%: 0.06%

# 4. Load instruction dataset
dataset = load_dataset("timdettmers/openassistant-guanaco")

# 5. Training configuration
training_args = TrainingArguments(
output_dir="./llama2-lora",
num_train_epochs=3,
per_device_train_batch_size=4,
gradient_accumulation_steps=4,
learning_rate=2e-4,
logging_steps=10,
save_strategy="epoch",
fp16=True, # Mixed precision
)

# 6. Train with SFTTrainer (Supervised Fine-Tuning)
trainer = SFTTrainer(
model=model,
train_dataset=dataset["train"],
args=training_args,
peft_config=lora_config,
dataset_text_field="text",
max_seq_length=512,
)

trainer.train()

# 7. Save only LoRA adapter (tiny!)
model.save_pretrained("./llama2-lora-adapter")

# 8. Load and use later
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained(model_name)
lora_model = PeftModel.from_pretrained(base_model, "./llama2-lora-adapter")

# Generate
inputs = tokenizer("Explain quantum computing:", return_tensors="pt")
outputs = lora_model.generate(**inputs, max_length=100)
print(tokenizer.decode(outputs[0]))

Resource Requirements:

  • GPU: RTX 3090 (24GB) or better
  • Time: 4-8 hours
  • Storage: Base model (14GB) + Adapter (50MB)

Key Observations:

  • Only 0.06% of parameters trained! (4M out of 6.7B)
  • Memory usage: ~18GB (vs ~90GB for full fine-tuning)
  • Can train multiple adapters and switch between them

Practice 3: Comparing LoRA Ranks​

Experiment: Train same model with different ranks and compare

# Train adapters with different ranks
ranks = [4, 8, 16, 32, 64]
results = {}

for r in ranks:
print(f"\n{'='*50}")
print(f"Training with rank r={r}")
print(f"{'='*50}")

lora_config = LoraConfig(
r=r,
lora_alpha=r*2, # Common practice: alpha = 2*r
target_modules=["q_proj", "v_proj"],
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM"
)

model = prepare_model_for_kbit_training(base_model)
model = get_peft_model(model, lora_config)

# Train and evaluate
trainer = SFTTrainer(...)
trainer.train()
metrics = trainer.evaluate()

results[r] = {
'trainable_params': sum(p.numel() for p in model.parameters() if p.requires_grad),
'loss': metrics['eval_loss'],
'adapter_size_mb': get_adapter_size(model)
}

# Compare results
import pandas as pd
df = pd.DataFrame(results).T
print(df)

# Expected output:
# trainable_params loss adapter_size_mb
# 4 2,097,152 2.45 8
# 8 4,194,304 2.31 16
# 16 8,388,608 2.24 32
# 32 16,777,216 2.21 64
# 64 33,554,432 2.19 128

Insights:

  • Rank 4-8: Fast, tiny, good for simple tasks
  • Rank 16-32: Sweet spot for most tasks
  • Rank 64+: Approaching full fine-tuning performance

Advanced Topics​

1. QLoRA (Quantized LoRA)​

What: Combines LoRA with 4-bit quantization for even lower memory usage.

Breakthrough: Train 65B parameter models on a single 48GB GPU!

from transformers import BitsAndBytesConfig

# QLoRA configuration
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_use_double_quant=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16
)

model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-2-70b-hf",
quantization_config=bnb_config,
device_map="auto"
)

# Now train with LoRA as before
# Memory usage: ~40GB for 70B model! 🎉

Use Case: Fine-tune massive models (65B, 70B+) on consumer hardware.


2. AdaLoRA (Adaptive LoRA)​

What: Dynamically adjusts rank during training based on importance.

Benefit: Automatically finds optimal rank per layer.

from peft import AdaLoraConfig, get_peft_model

adalora_config = AdaLoraConfig(
init_r=12, # Initial rank
target_r=8, # Target rank
tinit=200, # Warmup steps
tfinal=1000, # Total steps
delta_t=10, # Rank adjustment frequency
beta1=0.85,
beta2=0.85,
orth_reg_weight=0.5,
lora_alpha=32,
lora_dropout=0.1,
target_modules=["q_proj", "v_proj"],
task_type="CAUSAL_LM"
)

model = get_peft_model(model, adalora_config)

Result: Better performance than fixed-rank LoRA with similar parameter count.


3. The "Intruder Dimensions" Problem​

Recent Discovery (MIT Research, 2024):

LoRA introduces high-ranking singular vectors not present in full fine-tuning. These "intruder dimensions":

  • Reduce out-of-distribution generalization by 5-8%
  • Cause more forgetting in continual learning
  • Appear more with lower ranks (r ≤ 16)

Visualization:

Full Fine-Tuning Weight Matrix SVD:
Singular Values: [100, 95, 90, 85, 80, ...] ← Smooth decay
Singular Vectors: All similar to pre-trained

LoRA Weight Matrix SVD:
Singular Values: [100, 95, 90, 85, 80, ..., 15, 14, 13] ← Similar
↓
[50, 45, 40] ← INTRUDER!
Singular Vectors: Some very different from pre-trained

Solutions:

  1. Use higher rank (r=64+) with rank stabilization
  2. Use "rank-stabilized LoRA" techniques
  3. For critical applications, prefer full fine-tuning

4. Multi-Adapter Architecture​

Pattern: One base model + dynamic adapter loading

# Training phase: Train multiple adapters
adapters = {
'medical': train_lora(base_model, medical_data, r=16),
'legal': train_lora(base_model, legal_data, r=16),
'finance': train_lora(base_model, finance_data, r=16),
}

# Save adapters
for name, adapter in adapters.items():
adapter.save_pretrained(f"./adapters/{name}")

# Inference phase: Dynamically load adapters
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained("llama-2-7b")

# For medical query
medical_model = PeftModel.from_pretrained(base_model, "./adapters/medical")
response = medical_model.generate(...)

# Switch to legal
legal_model = PeftModel.from_pretrained(base_model, "./adapters/legal")
response = legal_model.generate(...)

# Or merge multiple adapters!
from peft import PeftModel
multi_model = PeftModel.from_pretrained(base_model, "./adapters/medical")
multi_model.load_adapter("./adapters/legal", adapter_name="legal")
multi_model.set_adapter(["medical", "legal"]) # Use both!

Decision Framework​

The Complete Decision Tree​

START: Do you need to fine-tune a large language model?
│
├─→ Q1: What's your budget?
│ ├─→ < $1,000: → LoRA (or QLoRA)
│ ├─→ $1,000 - $10,000: → LoRA (consider full if single critical task)
│ └─→ > $10,000: → Consider full fine-tuning
│
├─→ Q2: What GPU do you have?
│ ├─→ Consumer (RTX 3090/4090): → LoRA or QLoRA
│ ├─→ Single A100 (40-80GB): → LoRA or Full (for <7B models)
│ └─→ GPU Cluster (8+ A100s): → Full fine-tuning possible
│
├─→ Q3: How many tasks?
│ ├─→ 1 task: → Either (depends on other factors)
│ ├─→ 2-5 tasks: → LoRA (multi-adapter)
│ └─→ 5+ tasks: → Definitely LoRA
│
├─→ Q4: Accuracy requirements?
│ ├─→ >98% required: → Full fine-tuning
│ ├─→ 95-98%: → LoRA (high rank, r=64+)
│ └─→ <95%: → LoRA (standard rank, r=8-16)
│
├─→ Q5: Deployment constraints?
│ ├─→ Edge devices: → LoRA (smaller)
│ ├─→ Cloud: → Either
│ └─→ Need rapid updates: → LoRA
│
└─→ Q6: Continual learning needed?
├─→ Yes, multiple sequential tasks: → High-rank LoRA or Full
├─→ No: → Standard LoRA
└─→ Critical to avoid forgetting: → Full fine-tuning

Quick Selection Matrix​

Your SituationRecommended MethodRank/Config
Startup, limited GPU, 3+ tasksLoRAr=8-16
Enterprise, single critical taskFull Fine-TuningAll parameters
Research, multiple experimentsLoRAr=16-32
>100B model, limited hardwareQLoRAr=8-16, 4-bit
Medical/Legal high-stakesFull Fine-TuningAll parameters
Multi-lingual deploymentLoRA multi-adapterr=16 per language
Rapid A/B testingLoRAr=8 (fast training)
Edge device deploymentQLoRAr=4-8, quantized

Best Practices & Tips​

For LoRA Fine-Tuning​

  1. Start Small, Scale Up

    # Experiment progression
    1. Try r=8 first (fast, good baseline)
    2. If underfitting, increase to r=16
    3. For complex tasks, try r=32 or r=64
    4. Monitor validation loss to find sweet spot
  2. Target the Right Modules

    # For most language models
    target_modules = ["q_proj", "v_proj"] # Minimum (queries & values)

    # For better performance
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj"] # All attention

    # For maximum adaptation (more params)
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
    "gate_proj", "up_proj", "down_proj"] # Attention + FFN
  3. Learning Rate Selection

    # LoRA typically needs higher LR than full fine-tuning
    Full Fine-Tuning: lr = 1e-5 to 5e-5
    LoRA: lr = 1e-4 to 3e-4 (10x higher!)

    # Rule of thumb
    lora_lr = full_finetuning_lr * 10
  4. Rank Selection Guide

    Task Complexity → Recommended Rank
    ─────────────────────────────────────────
    Simple (sentiment) → r=4-8
    Medium (summarization) → r=8-16
    Complex (reasoning) → r=16-32
    Very complex (coding) → r=32-64
    Approaching full FT → r=128-256
  5. Prevent Intruder Dimensions

    # Use rank stabilization
    lora_config = LoraConfig(
    r=32,
    lora_alpha=64,
    use_rslora=True, # Rank-stabilized LoRA
    # ... other params
    )

For Full Fine-Tuning​

  1. Gradient Checkpointing (Save Memory)

    model.gradient_checkpointing_enable()
    # Trades computation for memory
    # Enables training larger models on same GPU
  2. Mixed Precision Training

    training_args = TrainingArguments(
    fp16=True, # For older GPUs
    bf16=True, # For A100s (better for large models)
    # Reduces memory usage by 2x
    )
  3. Distributed Training

    # Use DeepSpeed for multi-GPU
    training_args = TrainingArguments(
    deepspeed="ds_config.json",
    per_device_train_batch_size=1,
    gradient_accumulation_steps=16,
    )

    # ds_config.json
    {
    "train_batch_size": "auto",
    "zero_optimization": {
    "stage": 3, # ZeRO-3: Shard everything
    }
    }
  4. Prevent Catastrophic Forgetting

    # Use lower learning rate
    learning_rate = 1e-5 # vs 2e-4 for pre-training

    # Freeze some layers (partial fine-tuning)
    for i, layer in enumerate(model.layers):
    if i < 20: # Freeze first 20 layers
    for param in layer.parameters():
    param.requires_grad = False

    # Use elastic weight consolidation (EWC)
    # Add regularization term penalizing large weight changes

Common Mistakes & How to Avoid Them​

Mistake 1: Using LoRA rank that's too low​

❌ Problem: r=4 for complex instruction following

lora_config = LoraConfig(r=4, ...) # Too small!
# Result: Model can't learn complex patterns, poor performance

✅ Solution: Start with r=16, increase if needed

lora_config = LoraConfig(r=16, lora_alpha=32, ...)
# Monitor validation loss, increase to r=32 if plateauing

Mistake 2: Wrong learning rate​

❌ Problem: Using same LR for LoRA as full fine-tuning

# Full fine-tuning LR used for LoRA
training_args = TrainingArguments(learning_rate=2e-5, ...)
# Result: Slow convergence, underfitting

✅ Solution: Use 10x higher LR for LoRA

training_args = TrainingArguments(learning_rate=2e-4, ...)
# LoRA adapters need stronger signals to learn

Mistake 3: Not quantizing when memory is tight​

❌ Problem: Trying to load 7B model in FP16 on 16GB GPU

model = AutoModelForCausalLM.from_pretrained("llama-2-7b")
# Result: CUDA Out of Memory error

✅ Solution: Use 4-bit/8-bit quantization

model = AutoModelForCausalLM.from_pretrained(
"llama-2-7b",
load_in_4bit=True,
device_map="auto"
)
# Result: Fits in 8GB GPU!

Mistake 4: Training on too few examples​

❌ Problem: Fine-tuning on 100 examples

dataset = load_dataset(...)[:100] # Only 100 examples
# Result: Overfitting, poor generalization

✅ Solution: Minimum dataset sizes

Full Fine-Tuning: 10,000+ examples (ideally 100k+)
LoRA: 1,000+ examples (can work with 500 for simple tasks)
QLoRA: 500+ examples (most efficient)

If you have <500 examples:
→ Use prompt engineering or few-shot learning instead

Mistake 5: Not saving checkpoints​

❌ Problem: Training crashes after 20 hours, no checkpoints

training_args = TrainingArguments(
save_strategy="no", # Dangerous!
)

✅ Solution: Regular checkpointing

training_args = TrainingArguments(
save_strategy="steps",
save_steps=500,
save_total_limit=3, # Keep last 3 checkpoints
)

Real Cost Comparison​

Example: Fine-tuning LLaMA-2-7B for 3 epochs on 10,000 examples​

Full Fine-Tuning:

Hardware: 4× A100 (80GB) via AWS
Cost: $32.77/hour per GPU
Duration: 48 hours
Total: $32.77 × 4 × 48 = $6,293

Storage: 14GB per model
If 5 tasks: 70GB total

Total Investment: ~$6,300 + ongoing storage

LoRA (r=16):

Hardware: 1× RTX 4090 (24GB) - owned or $2/hour cloud
Cost: $2/hour
Duration: 12 hours
Total: $2 × 12 = $24

Storage: 14GB base + (5 × 50MB adapters) = 14.25GB

Total Investment: ~$24 + one-time GPU purchase (~$1,600)

Savings: 99.6% 🎉


1. LoRA+ and LoRA-FA​

  • LoRA+: Different learning rates for A and B matrices
  • LoRA-FA: Frozen-A variant (only train B matrix)
  • Result: Faster convergence, better performance

2. DoRA (Weight-Decomposed LoRA)​

  • Decomposes weights into magnitude and direction
  • Bridges gap between LoRA and full fine-tuning
  • 20% better performance on reasoning tasks

3. MultiLoRA & LoRA Composition​

  • Train multiple LoRA adapters
  • Compose them at inference time
  • Example: (medical_adapter × 0.7) + (general_adapter × 0.3)

4. LoRA for Vision & Multimodal Models​

  • VoRA: LoRA for vision transformers
  • Successful for CLIP, Stable Diffusion fine-tuning
  • 90% memory reduction for image model fine-tuning

Summary & Recommendations​

The Bottom Line​

LoRA is the default choice for most practitioners in 2024-2025.

Reasons:

  • ✅ 99% cost reduction
  • ✅ Accessible on consumer hardware
  • ✅ Fast iteration (hours vs days)
  • ✅ Easy multi-task deployment
  • ✅ Good enough performance for most applications (95-99% of full)

Full Fine-Tuning is for special cases:

  • High-stakes applications (medical, legal, financial)
  • Unlimited budget and resources
  • Need absolute best performance
  • Research settings exploring model limits

Your Action Plan​

Week 1: Start with LoRA

1. Choose base model (LLaMA-2-7B, Mistral-7B, Qwen)
2. Prepare 1,000+ examples
3. Train LoRA (r=16) on 1× GPU
4. Evaluate on validation set
5. Iterate on rank and hyperparameters

Week 2: Optimize

1. Experiment with ranks: 8, 16, 32
2. Try different target modules
3. Test QLoRA if memory constrained
4. A/B test different adapters

Month 2+: Scale if Needed

if performance < requirements:
if budget_available:
try full fine-tuning
else:
try higher rank LoRA (r=64-128)
or try DoRA/LoRA+ variants

Additional Resources​

Code Repositories​

Tutorials​

Communities​


Remember:

  • Start with LoRA (r=16)
  • Scale to full fine-tuning only if truly needed
  • Focus on data quality over method choice
  • Iterate quickly and measure results

The best method is the one that:

  1. Fits your budget
  2. Meets your accuracy requirements
  3. Enables fast iteration
  4. Scales with your needs

AI LLM Tool Ollama: Architecture and Conversation Processing Flow Analysis

· 阅读需 4 分钟

Overview of Ollama​

Ollama is a convenient tool for running LLM (Large Language Models) quickly. With Ollama, users can easily interact with large language models without complex environment configurations.

This article will analyze the overall architecture of Ollama and detail the specific processing flow when users engage in conversations with Ollama.

Ollama Architecture​

Ollama Architecture

Ollama employs a classic CS (Client-Server) architecture, where:

  • Client: Interacts with the user via the command line.
  • Server: Can be started via the command line, desktop application (based on the Electron framework), or Docker. Regardless of the startup method, the same executable file is ultimately invoked.
  • Communication: Client and Server communicate using HTTP.

The Ollama Server consists of two core components:

  • ollama-http-server: Handles interactions with the client.
  • llama.cpp: Serves as the LLM inference engine, responsible for loading and running large language models, processing inference requests, and returning results.
  • Interaction: ollama-http-server and llama.cpp also communicate via HTTP.

Note: llama.cpp is an independent open-source project that is cross-platform and hardware-friendly, capable of running on devices without GPUs, including Raspberry Pi.

Ollama Storage Structure​

Ollama Storage Architecture

Ollama uses the default local storage folder path $HOME/.ollama. The file structure is as follows:

$HOME/.ollama/
├── blobs/
├── manifests/
├── logs/
│ └── server.log
├── history
├── id_ed25519
└── id_ed25519.pub

Files can be categorized into three types:

  • Log Files: Includes the history file, which records user conversation inputs, and the logs/server.log server log file.
  • Key Files: Includes the id_ed25519 private key and id_ed25519.pub public key.
  • Model Files: Includes blobs raw data files and manifests metadata files.

The metadata files, such as models/manifests/registry.ollama.ai/library/llama3.2/latest, contain the following content:

llama3.2-latest

{
"digest": "sha256:abcdef1234567890",
"size": 123456789,
"annotations": {
"org.opencontainers.image.title": "llama3.2",
"org.opencontainers.image.version": "latest"
}
}

As shown above, manifests files are in JSON format and draw inspiration from the OCI spec (Open Container Initiative Specification) in cloud-native and container domains. The digest field in manifests corresponds to blobs.

Ollama Conversation Processing Flow​

Ollama Conversation Processing Flow

The general flow of a user conversation with Ollama is as follows:

  1. User Initiates Conversation: The user executes the command ollama run llama3.2 via the CLI to start a conversation (llama3.2 is an open-source large language model; you can also use other LLMs).

  2. Preparation Phase:

    • The CLI client sends an HTTP request to ollama-http-server to fetch model information. The server attempts to read the local manifests metadata files. If not found, it responds with a 404 not found.
    • When the model does not exist, the CLI client requests ollama-http-server to pull the model from the remote repository to the local storage.
    • The CLI client then requests the model information again.
  3. Interactive Conversation Phase:

    • The CLI sends an empty message /api/generate request to ollama-http-server, which internally handles some channel processing (using Go's channels).
    • If the model information contains messages, they are printed. Users can save the current model and session conversation history as a new model, and the conversation history is saved as messages.
    • The conversation officially begins: The CLI calls the /api/chat interface to request ollama-http-server, which relies on the llama.cpp engine to load the model and perform inference (llama.cpp also serves as an HTTP server). ollama-http-server first sends a /health request to llama.cpp to confirm its health status, then sends a /completion request to obtain the conversation response, and finally returns it to the CLI for display.

Through the above steps, Ollama completes the interaction between the user and the large language model.

Conclusion​

Ollama integrates the llama.cpp inference engine and further encapsulates it, making complex LLM technology accessible. It provides developers and technical personnel with an efficient and flexible tool, effectively supporting large language model inference and interaction in various application scenarios.

References:​

标签:

Introduction to 5 AI Generation Models

· 阅读需 7 分钟

Introduction to 5 AI Generation Models

In this article, we will explore some of the most common AI models used for generating data, images, or other types of content. These models use deep learning techniques and are called generative models because they create new data based on the patterns they have learned. We will cover the following models:


1. Variational Autoencoders (VAE)​

VAE is a type of generative model that learns to compress data into a smaller format and then decompress it back to its original form. Unlike a normal autoencoder, VAE introduces randomness, allowing it to generate new data that looks like the original data but is slightly different.

How VAE Works:​

  1. Encoder: Compresses the input data into a smaller "latent space."
  2. Decoder: Reconstructs the data from the latent space.
  3. Variational: Adds randomness (a probability distribution) to the latent space, allowing the model to generate new data.

Example Diagram:​

VAE Diagram

+-------------------+
| Input Data |
| x |
+-------------------+
|
v
+-------------------+
| Encoder |
| (Neural Network) |
| |
| Outputs: |
| - μ (Mean) |
| - log σ² (Log |
| Variance) |
+-------------------+
|
v
+-------------------+
| Reparameterization|
| Trick: |
| z = μ + σ * ε |
| (ε ~ N(0, I)) |
+-------------------+
|
v
+-------------------+
| Decoder |
| (Neural Network) |
| |
| Outputs: |
| - Reconstructed |
| Data x' |
+-------------------+
|
v
+-------------------+
| Loss Function |
| |
| - Reconstruction |
| Loss |
| - KL Divergence |
+-------------------+

Python Code Example:​

import tensorflow as tf
from tensorflow.keras import layers

# Encoder
input_img = tf.keras.Input(shape=(28, 28, 1))
x = layers.Flatten()(input_img)
x = layers.Dense(256, activation='relu')(x)
z_mean = layers.Dense(2)(x)
z_log_var = layers.Dense(2)(x)

# Sampling layer
def sampling(args):
z_mean, z_log_var = args
batch = tf.shape(z_mean)[0]
dim = tf.shape(z_mean)[1]
epsilon = tf.keras.backend.random_normal(shape=(batch, dim))
return z_mean + tf.exp(0.5 * z_log_var) * epsilon

z = layers.Lambda(sampling)([z_mean, z_log_var])

# Decoder
decoder_input = layers.Input(shape=(2,))
x = layers.Dense(256, activation='relu')(decoder_input)
x = layers.Dense(28 * 28, activation='sigmoid')(x)
output_img = layers.Reshape((28, 28, 1))(x)

# Create VAE model
vae = tf.keras.Model(input_img, output_img)

2. Generative Adversarial Networks (GAN)​

GAN is a generative model consisting of two parts: a generator and a discriminator. The generator tries to create fake data, and the discriminator tries to detect whether the data is real or fake. Over time, the generator gets better at fooling the discriminator.

How GAN Works:​

  1. Generator: Creates fake data.
  2. Discriminator: Tries to differentiate between real and fake data.
  3. Adversarial Training: The two networks compete, helping the generator improve.

Example Diagram:​

GAN Diagram

+-------------------+
| Random Noise |
| z |
+-------------------+
|
v
+-------------------+
| Generator |
| (Neural Network) |
| |
| Outputs: |
| - Generated Data |
| G(z) |
+-------------------+
|
v
+-------------------+
| Discriminator |
| (Neural Network) |
| |
| Inputs: |
| - Real Data x |
| - Generated Data |
| G(z) |
| |
| Outputs: |
| - Probability |
| Score (0 to 1) |
+-------------------+
|
v
+-------------------+
| Loss Function |
| |
| - Generator Loss |
| - Discriminator |
| Loss |
+-------------------+

Python Code Example:​

import tensorflow as tf
from tensorflow.keras import layers

# Generator
def build_generator():
model = tf.keras.Sequential([
layers.Dense(256, activation='relu', input_dim=100),
layers.Dense(512, activation='relu'),
layers.Dense(1024, activation='relu'),
layers.Dense(28 * 28, activation='sigmoid'),
layers.Reshape((28, 28, 1))
])
return model

# Discriminator
def build_discriminator():
model = tf.keras.Sequential([
layers.Flatten(input_shape=(28, 28, 1)),
layers.Dense(512, activation='relu'),
layers.Dense(256, activation='relu'),
layers.Dense(1, activation='sigmoid')
])
return model

# Instantiate models
generator = build_generator()
discriminator = build_discriminator()

# Create GAN model
gan_input = tf.keras.Input(shape=(100,))
generated_img = generator(gan_input)
discriminator.trainable = False
validity = discriminator(generated_img)
gan = tf.keras.Model(gan_input, validity)

3. Autoregressive Models (AR)​

Autoregressive Models predict the next data point based on previous data points. For example, in language models like Transformers, the model predicts the next word in a sentence based on the words that came before it.

How AR Works:​

  • It uses past information to predict future information.
  • Often used in natural language processing (NLP) and time series data.

Example Diagram (Transformer):​

Transformer

+-------------------+
| Input Sequence |
| x |
+-------------------+
|
v
+-------------------+
| Positional |
| Encoding |
+-------------------+
|
v
+-------------------+
| Transformer |
| Encoder |
| |
| - Multi-Head |
| Attention |
| - Feed-Forward |
| Layers |
+-------------------+
|
v
+-------------------+
| Output Layer |
| |
| - Softmax |
+-------------------+
|
v
+-------------------+
| Predicted Token |
| y |
+-------------------+

Python Code Example (Using AR):​

import tensorflow as tf

# Example of a simple autoregressive model for time series data
model = tf.keras.Sequential([
layers.Input(shape=(10, 1)),
layers.SimpleRNN(50, return_sequences=True),
layers.Dense(1)
])

model.compile(optimizer='adam', loss='mean_squared_error')

# Dummy data
import numpy as np
X = np.random.rand(1000, 10, 1)
y = np.random.rand(1000, 10, 1)

# Train the AR model
model.fit(X, y, epochs=10)

4. Flow Models​

Flow Models are generative models that work by learning how to reverse a complex transformation back to a simple distribution, like a Gaussian distribution. They are designed to model complex data distributions.

How Flow Models Work:​

  • They transform simple data distributions (e.g., Gaussian) into complex ones.
  • They are invertible, meaning the process can be reversed easily.

Example Diagram:​

Flow Diagram

+-------------------+
| Simple |
| Distribution |
| z |
+-------------------+
|
v
+-------------------+
| Flow Model |
| |
| - Coupling Layers|
| - Invertible |
| Transformations|
+-------------------+
|
v
+-------------------+
| Complex |
| Distribution |
| x |
+-------------------+

Python Code Example:​

import tensorflow as tf
import tensorflow_probability as tfp

# Simple Normalizing Flow example
tfd = tfp.distributions

base_dist = tfd.MultivariateNormalDiag(loc=[0., 0.])
flow = tfp.bijectors.RealNVP(num_masked=1)

# Transformed distribution
transformed_dist = tfd.TransformedDistribution(distribution=base_dist, bijector=flow)

# Sample from the flow model
samples = transformed_dist.sample(1000)
print(samples)

5. Diffusion Models​

Diffusion Models generate data by slowly adding random noise to a dataset and then learning to reverse the process. These models are great for creating realistic images from noise.

How Diffusion Models Work:​

  1. Noise is added to the data step by step.
  2. The model learns to reverse this process, removing the noise and recovering the original data.

Example Diagram:​

Diffusion Models

+-------------------+
| Real Data |
| x |
+-------------------+
|
v
+-------------------+
| Forward |
| Diffusion |
| Process |
| |
| - Add Noise |
| - Gaussian Noise |
+-------------------+
|
v
+-------------------+
| Noisy Data |
| x_t |
+-------------------+
|
v
+-------------------+
| Reverse |
| Diffusion |
| Process |
| |
| - Denoise |
| - Neural Network |
+-------------------+
|
v
+-------------------+
| Generated Data |
| x_0 |
+-------------------+

Python Code Example:​

import torch
import torch.nn as nn

class DiffusionModel(nn.Module):
def __init__(self):
super(DiffusionModel, self).__init__()
self.fc = nn.Linear(28*28, 28*28)

def forward(self, x):
noise = torch.randn_like(x)
return x + noise

# Dummy diffusion process
model = DiffusionModel()

# Example data (images flattened)
X = torch.randn(64, 28*28)

# Adding noise through the model
noisy_X = model(X)
print(noisy_X)

Conclusion​

Each of these models serves a different purpose in the field of AI. VAE and GAN are great for image generation, while AR models excel in sequential data like language or time series. Flow and Diffusion models are newer but powerful tools for generating complex, realistic data. Learning about these models can help you understand how machines can create art, music, text, and more!

Key Takeaways:​

  • VAE: Generates new data by introducing randomness in the latent space.
  • GAN: Uses adversarial training to improve the quality of generated data.
  • AR: Predicts future data points based on past information.
  • Flow Models: Transform simple distributions into complex ones using invertible transformations.
  • Diffusion Models: Generate data by reversing a diffusion process that adds noise.

By mastering these techniques, you can build robust and creative AI systems that generate high-quality content.

标签:

Introduction to Regression Algorithms

· 阅读需 6 分钟

Regression is a type of analysis used to understand relationships between variables. In machine learning, regression algorithms predict continuous outcomes based on input data. In this article, we will cover several common regression algorithms with sample Python code snippets and visual aids to help you understand how they work. Let's dive in!


Index​


1. Linear Regression​

Linear Regression models the relationship between input (x) and output (y) as a straight line. It tries to minimize the difference between the predicted and actual values.

Example Diagram:​

Linear Regression

Python Code Example:​

from sklearn.linear_model import LinearRegression
import numpy as np

# Sample data
X = np.array([[1], [2], [3], [4], [5]])
y = np.array([2, 4, 5, 4, 5])

# Create and train model
model = LinearRegression()
model.fit(X, y)

# Predict new values
y_pred = model.predict(np.array([[6]]))
print(f"Prediction for X=6: {y_pred[0]}")

2. Logistic Regression​

Logistic Regression is used for binary classification, predicting probabilities that the input belongs to one of two classes (e.g., "yes" or "no").

Example Diagram:​

Logistic Regression

Python Code Example:​

from sklearn.linear_model import LogisticRegression

# Sample data (1 or 0 outcome)
X = np.array([[1], [2], [3], [4], [5]])
y = np.array([0, 0, 0, 1, 1])

# Create and train model
model = LogisticRegression()
model.fit(X, y)

# Predict new value
y_pred = model.predict(np.array([[3.5]]))
print(f"Prediction for X=3.5: {y_pred[0]}")

3. Decision Tree Regression​

Decision Tree Regression splits the data into different regions based on decisions and forms a tree structure. Each leaf node represents a prediction.

Example Diagram:​

Decision Tree

Python Code Example:​

from sklearn.tree import DecisionTreeRegressor

# Sample data
X = np.array([[1], [2], [3], [4], [5]])
y = np.array([1.5, 2.5, 3.0, 3.5, 4.0])

# Create and train model
model = DecisionTreeRegressor()
model.fit(X, y)

# Predict new values
y_pred = model.predict(np.array([[6]]))
print(f"Prediction for X=6: {y_pred[0]}")

4. Random Forest Regression​

Random Forest Regression is an ensemble of multiple decision trees that makes predictions by averaging the results of the trees.

Example Diagram:​

Random Forest

Python Code Example:​

from sklearn.ensemble import RandomForestRegressor

# Sample data
X = np.array([[1], [2], [3], [4], [5]])
y = np.array([2, 4, 6, 8, 10])

# Create and train model
model = RandomForestRegressor(n_estimators=100)
model.fit(X, y)

# Predict new value
y_pred = model.predict(np.array([[6]]))
print(f"Prediction for X=6: {y_pred[0]}")

5. Support Vector Regression (SVR)​

Support Vector Regression (SVR) fits the best line within a margin of error, known as the epsilon tube, focusing on points within this margin.

Example Diagram:​

SVR

Python Code Example:​

from sklearn.svm import SVR

# Sample data
X = np.array([[1], [2], [3], [4], [5]])
y = np.array([1, 2, 3, 5, 8])

# Create and train model
model = SVR(kernel='linear')
model.fit(X, y)

# Predict new value
y_pred = model.predict(np.array([[6]]))
print(f"Prediction for X=6: {y_pred[0]}")

6. K-Nearest Neighbors (KNN) Regression​

KNN Regression finds the k-nearest data points to the input and predicts the value based on their average.

Example Diagram:​

KNN

Python Code Example:​

from sklearn.neighbors import KNeighborsRegressor

# Sample data
X = np.array([[1], [2], [3], [4], [5]])
y = np.array([1, 2, 1.5, 3.5, 2])

# Create and train model
model = KNeighborsRegressor(n_neighbors=2)
model.fit(X, y)

# Predict new value
y_pred = model.predict(np.array([[6]]))
print(f"Prediction for X=6: {y_pred[0]}")

7. Bayesian Linear Regression​

Bayesian Linear Regression adds probability to the linear regression model, providing a measure of uncertainty in predictions.

Python Code Example:​

from sklearn.linear_model import BayesianRidge

# Sample data
X = np.array([[1], [2], [3], [4], [5]])
y = np.array([1.2, 2.3, 3.1, 4.8, 5.5])

# Create and train model
model = BayesianRidge()
model.fit(X, y)

# Predict new value
y_pred = model.predict(np.array([[6]]))
print(f"Prediction for X=6: {y_pred[0]}")

8. Lasso and Ridge Regression​

Lasso Regression and Ridge Regression add regularization to reduce overfitting in linear regression. Lasso uses L1 regularization (shrinking coefficients to zero), while Ridge uses L2 regularization.

Lasso Regression Code Example:​

from sklearn.linear_model import Lasso

# Sample data
X = np.array([[1], [2], [3], [4], [5]])
y = np.array([2, 3, 4, 5, 6])

# Create and train model
model = Lasso(alpha=0.1)
model.fit(X, y)

# Predict new value
y_pred = model.predict(np.array([[6]]))
print(f"Prediction for X=6: {y_pred[0]}")

Ridge Regression Code Example:​

from sklearn.linear_model import Ridge

# Create and train model
model = Ridge(alpha=1.0)
model.fit(X, y)

# Predict new value
y_pred = model.predict(np.array([[6]]))
print(f"Prediction for X=6: {y_pred[0]}")

9. Regression Analysis​

Regression Analysis helps understand the relationships between variables, often used in fields such as economics and biology.


10. Neural Networks with Regression​

Neural Networks can capture complex relationships between variables. For regression, the output layer of the network predicts continuous values.

Example Diagram:​

Neural Networks

Python Code Example (Using Keras):​

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

# Sample data
X = np.array([[1], [2], [3], [4], [5]])
y = np.array([1, 2, 3, 4, 5])

# Create neural network model
model = Sequential()
model.add(Dense(10, input_dim=1, activation='relu'))
model.add(Dense(1)) # Output layer

# Compile and train model
model.compile(optimizer='adam', loss='mean_squared_error')
model.fit(X, y, epochs=100, verbose=0)

# Predict new value
y_pred = model.predict(np

.array([[6]]))
print(f"Prediction for X=6: {y_pred[0][0]}")

Conclusion​

Each regression algorithm has its strengths and is suited for different tasks. Linear regression works well for simple data, while Random Forests and Neural Networks are ideal for more complex datasets. Learning and experimenting with these algorithms will help you solve a variety of predictive problems.

标签: