Skip to main content

2 posts tagged with "chatgpt"

View All Tags

Building a ChatGPT-like System - II

· 5 min read

This guide provides an in-depth exploration of building a ChatGPT-style conversational AI system. We cover everything from model setup and inference engine design to system architecture, incorporating caching, scalability, and production-ready features.

System Overview​

A conversational AI system has several core components responsible for processing, generating, and managing dialogue flow with users. This overview explains the minimal structure for a functional, extensible ChatGPT-style system.

1. Core Dialogue Architecture​

The architecture focuses on the interaction between the user, the web interface, and the inference engine that generates responses.

Diagram: Core Dialogue Flow​

+---------------+ +-------------------+ +-----------------------+
| Web UI | <-- HTTP --> | Server/API | <-- REST --> | Inference Runtime |
| (User Access) | | (Request Routing) | | (LLM Model Handling) |
+---------------+ +-------------------+ +-----------------------+
^ |
|<--------------------------------------------------------+
  1. Web UI: Handles user interaction and displays responses.
  2. Server/API: Receives requests and forwards them to the inference runtime.
  3. Inference Runtime: Loads the large language model (LLM), processes the input, and generates a response.

2. Session and History Management​

Problem of Stateless Inference​

Standard inference runtimes are typically stateless, meaning they cannot recall previous messages in a conversation. To handle multi-turn dialogue, we integrate a database to persist session data and user message history.

Diagram: Enhanced System with Session and History Management​

+---------------+ +-------------------+ +-----------------------+
| Web UI | <-- HTTP --> | Server/API | <-- REST --> | Inference Runtime |
| | | | | |
+---------------+ +-------------------+ +-----------------------+
|
+---------------- Database ----------------+
| User Session, ChatID, and Messages |
+------------------------------------------+
  1. Database: Stores user sessions, chat histories, and other metadata.
  2. Server/API: Uses REST endpoints to interact with the database, retrieving history when needed.

3. REST API Design​

The Server/API manages communication with the Web UI via REST APIs, handling user requests, responses, and session tracking.

REST Endpoints:

  • POST /chat: Starts a new conversation.
  • POST /chat/:chatID/completion: Continues a conversation in an existing session.
  • GET /chats: Retrieves the list of active sessions.
  • DELETE /chat/:chatID: Deletes a specified session.

Sample API Call to Start a Conversation

curl -X POST http://yourserver.com/chat \
-H "Content-Type: application/json" \
-d '{"userID": "12345", "message": "Hello, ChatGPT!"}'

4. Message Structure and Prompt Format​

The prompt format structures conversation history and provides context to the model.

Example Prompt Format in JSON

[
{ "role": "system", "content": "You are a helpful assistant." },
{ "role": "user", "content": "Hello!" },
{ "role": "assistant", "content": "Hello there, how may I assist you today?" },
{ "role": "user", "content": "How are you?" }
]
  1. Role: Indicates the message source (system, user, or assistant).
  2. Content: The message text.

5. Managing History and Context​

Techniques for History Management​

  • Direct Prompt Fill: Append all previous messages to the prompt, suitable for short interactions.
  • Context Truncation: Omit older messages when context length exceeds the model’s token limit.
  • History Summarization: Generate a summary of prior messages to condense the context, enabling longer conversations within token limits.

6. System Extensions and Scalability​

In a high-traffic environment, it is essential to scale the system horizontally and incorporate caching for repeated requests.

7. Caching Strategies​

Implementing caching reduces redundant processing for similar queries. This approach involves caching semantic meanings of requests rather than exact matches.

Diagram: Cache with Vector Embeddings for Similar Query Detection​

+---------------+ +------------+ +----------------+
| User Query | -> Embedding -> | Cache | Cache Hit -> | Response |
+---------------+ +------------+ +----------------+
| No Hit
v
+---------------------------+ +--------------------+
| Text Embedding Model | --> | Vector Storage and |
| (e.g., BERT/Word2Vec) | | Search Database |
+---------------------------+ +--------------------+
  1. Embedding Runtime: Converts queries into vector embeddings.
  2. Vector Storage: Stores embeddings to detect semantically similar queries.

Cache Scope Considerations​

  • Single User Cache: Stores responses per user session.
  • Global Cache: Shared across all users, introducing potential privacy considerations.

8. Elastic Scaling and Load Balancing​

Scaling is crucial for supporting high traffic. Stateless components like the server and inference runtime can be scaled horizontally using Kubernetes.

Diagram: Load Balanced System Architecture with Elastic Scaling​

+--------------------------+ +---------------+ +--------------------------+
| Gateway | ---> | Server | ---> | Inference |
| | | | | Runtime (Auto-scalable) |
| Load Balancer | | (Stateless) | | |
+--------------------------+ +---------------+ +--------------------------+
  • Gateway: Distributes traffic across multiple servers.
  • Message Queue (MQ): Handles requests asynchronously for stable inference.

9. Production Readiness​

To make the system production-ready, additional steps are essential.

Checklist for Production​

  • Technology Selection:
    • Database: PostgreSQL or MongoDB for session persistence.
    • Inference Engine: Use efficient models like llama.cpp, HuggingFace Transformers, or vLLM.
  • Observability:
    • Implement logging, tracing, and metrics.
  • CI/CD and Deployment:
    • Use Kubernetes or cloud-based deployment environments.

Conclusion​

This guide provides a foundational approach to building a ChatGPT-like system, managing sessions, scaling the architecture, and ensuring production reliability. From basic dialogue to advanced caching and scalability, these steps can help create a highly responsive and robust conversational AI.

Building a ChatGPT-like System

· 5 min read

Introduction​

ChatGPT is a conversational system based on Large Language Models (LLMs). This article introduces the process of building a system similar to ChatGPT, covering everything from the model, inference engine, to the overall architecture.


System Overview​

Our primary focus will be on the core of the conversation system.

System Diagram:

+--------------+
| Web |
| Interface |
+--------------+
|
|
v
+--------------+
| Server |
+--------------+
|
|
v
+--------------------------+
| Inference Runtime |
| (LLM Inference Engine) |
+--------------------------+
|
|
v
+--------------+
| Response |
+--------------+

Core Dialogue Flow

In this basic system framework:

  • The web interface interacts with users,
  • The server receives user requests and forwards them to the inference runtime (inference engine),
  • The inference runtime loads the LLM to generate a response, returning it to the user.

Session and Historical Message Management​

However, the above system has a critical flaw: it lacks session and historical message management for users. Common inference runtimes are typically stateless and do not directly support multi-turn conversations with historical context storage, meaning the system "forgets" previous exchanges within a single session.

To address this, we need some essential modifications.

Enhanced System Diagram:

+--------------+
| Web |
| Interface |
+--------------+
|
|
v
+--------------+
| Server |
+--------------+
|
|
v
+--------------------------+
| Inference Runtime |
| (LLM Inference Engine) |
+--------------------------+
|
|
v
+--------------+
| Database |
+--------------+

Enhanced System with Session ahdn History Management

In this setup, we introduce a database component to store user sessions and historical messages.

Communication Protocols and APIs​

For communication, the web interface and server can use HTTP protocols. Basic REST APIs might include:

  • POST /chat: Initiate a new session.
  • POST /chat/:chatID/completion: Continue a conversation within an existing session.
  • GET /chats: Retrieve a list of sessions.
  • DELETE /chat/:chatID: Delete a specific session.

The server persists data in the database, structuring conversation messages with essential fields like userID, chatID, userMessage, and assistantMessage.

Standardized Prompt Format​

When sending data to the inference runtime, the server uses a standardized prompt format:

[
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": "Hello!"
},
{
"role": "assistant",
"content": "Hello there, how may I assist you today?"
},
{
"role": "user",
"content": "How are you?"
}
]

The role field represents different participants: system defines the conversation context, user represents user input, and assistant represents the model’s output.

Managing Historical Messages​

There are several ways to handle historical messages:

  • Direct Prompt Filling: Add historical messages in the prompt as user and assistant entries. This works well for short conversations.
  • Dynamic Context Adjustment: Discard earlier messages to fit within the LLM’s token limit.
  • Message Summarization: Use the inference engine to summarize conversation history, compressing information before filling the prompt.

With session and historical message management, we establish a basic system framework.


System Expansion​

As the user base grows, the system will require further scalability.

Adding Cache Mechanism​

One approach to scalability is implementing a cache to avoid repeated inference processes. The cache key-value pair would be the user's question and the AI's response. A cache hit would depend on whether the semantic content of the question is similar. For example, if two questions have different wording but the same meaning, the system should return the same response.

Enhanced Caching System Diagram:

+--------------+
| Web |
| Interface |
+--------------+
|
|
v
+--------------+
| Server |
+--------------+
|
|
v
+--------------------------+
| Inference Runtime |
| (LLM Inference Engine) |
+--------------------------+
|
|
v
+--------------------------+
| Embedding |
| Runtime & Model |
+--------------------------+
|
|
v
+--------------------------+
| Vector Storage & Search |
+--------------------------+

Cache with Vector Embeddings for Similar Query Detection

Apart from the cache module, we add embedding runtime and a text embedding model to convert text into vectors. If two vectors are close, their text content is semantically similar. Vector storage and search modules are used to store and retrieve vectors.

Scope of Caching​

Caching can be scoped to:

  • Single-user cache: Caching value is low as users rarely repeat the same question.
  • Global cache: Allows reuse across users, but risks exposing sensitive data.

After data analysis and validation, decide if cache implementation is beneficial.

Elastic Scalability​

Elastic scaling is another key to handling high concurrency. The server is stateless, which allows for easy horizontal scaling.

Scalability System Diagram:

+--------------+
| Web |
| Interface |
+--------------+
|
|
v
+--------------+
| Gateway |
| (Load Bal.) |
+--------------+
|
|
v
+--------------------------+
| Server Instances |
+--------------------------+
|
|
v
+--------------------------+
| Inference Runtimes |
+--------------------------+

Load Balanced System Architecture with Elastic Scaling

Here, we introduce a gateway for load balancing. The inference runtime is also stateless and supports elastic scaling, but it requires more hardware resources and may respond slower than the server. To maintain stability under peak load, add a message queue (MQ) and change request handling to asynchronous, improving system resilience.


Production Readiness​

The system architecture covers the basic logic, but to achieve production-readiness, further steps are necessary:

  • Technology Selection: Choose a database (e.g., PostgreSQL or MongoDB) and inference engine (e.g., llama.cpp, HuggingFace Transformers, vLLM) and map logical components to physical components.
  • Observability: Integrate logging, tracing, metrics, and set up monitoring and alerting.
  • CI/CD and Deployment Environment: Configure automation pipelines for continuous integration and deployment, and select the appropriate environment (e.g., cloud platforms or Kubernetes).

Conclusion​

This article provides an overview of constructing a ChatGPT-like system, covering the core conversation flow and expanding to include session management and historical message storage. We discussed strategies for system expansion and laid out steps toward production readiness.


References​


This enhanced Markdown document covers each requested update, including detailed technical sections, diagrams, keywords, and sections for expansion and production readiness.