# Backend Overview (Fully Modularized)

The J4F Assistant backend is completely modularized with LangChain-powered streaming and clean separation of concerns.

## Developer & Migration Notes

### Migration to Modular Architecture
- The backend has been refactored to a fully modular architecture.
- All core functionalities are now in the `src/` directory, with clear separation of concerns.
- Please update any custom scripts or integrations to align with the new directory structure.

### LangChain Streaming Integration
- The assistant now uses **LangChain exclusively** for streaming.
- The `StreamingManager.js` module handles all streaming using LangChain.
- Legacy streaming code has been removed. Update any custom integrations to use the new LangChain-powered streaming.

### Configuration Changes
- All configuration is now managed via environment variables (`.env` file).
- Update your environment configuration to match the new structure.
- Feature flags are available to enable/disable specific features (e.g., file operations, system commands).

### Security Enhancements
- Security modules have been added to handle command execution, risk analysis, and audit logging.
- All sensitive operations must go through the appropriate security modules.
- Review and update any custom security implementations to align with the new modular security architecture.

### New Module/System Integration
- To add new tools or services, create a new module in the appropriate directory (`core/tools/`, `services/`, etc.).
- Register new modules in the corresponding orchestrator (e.g., `core/tools.js`).
- Update API documentation and wiki pages for any new features or modules.

## Directory Structure (2025)
- **src/core/**: Core logic modules (assistant, tools, security, streaming, prompt, knowledge)
- **src/interfaces/**: User interfaces (web, CLI, modular API routes)
- **src/services/**: Background services (scheduler, OpenWebUI integration)
- **src/utils/**: Utilities (logging, helpers)
- **data/**: Persistent data (conversations, workspaces)
- **logs/**: Log files (audit, assistant logs)

## Modular Architecture
- **Orchestrators**: High-level coordinators (assistant, tools, secure-executor)
- **Managers**: Specialized modules (context, model, streaming, knowledge, prompt, security)
- **Tool Modules**: File, system, knowledge, utility tools
- **Security Modules**: Command execution, risk analysis, audit logging, confirmation, workspace management
- **API Routes**: Modular route handlers for assistant, conversation, model, terminal, workspace

### Example: Modular Route Registration
```js
// src/interfaces/web-modular.js
import { registerChatRoutes } from './routes/assistant/ChatRoutes.js';
import { registerStreamingRoutes } from './routes/assistant/StreamingRoutes.js';
import { registerContextRoutes } from './routes/assistant/ContextRoutes.js';

export function registerAllRoutes(app) {
  registerChatRoutes(app);
  registerStreamingRoutes(app);
  registerContextRoutes(app);
  // ...other routes
}
```

## LangChain Streaming (Exclusive)
- All streaming is handled by LangChain-powered modules (no legacy fallback)
- Conversation memory, tool integration, and error recovery are built-in

## Event-Driven Backend
- Security, audit, and confirmation modules emit events for UI and logging
- Modular managers communicate via events and clean interfaces

## Core Modules

### Entry Point
- `main.js`: Application entry point, initializes all services and interfaces

### Core Orchestrators
- `core/assistant.js`: Main AI orchestrator, delegates to specialized managers
- `core/tools.js`: Tool system orchestrator, delegates to tool modules
- `core/secure-executor.js`: Security orchestrator, manages security workflow

### Specialized Managers
- `core/context/ContextManager.js`: Conversation and context management
- `core/model/ModelManager.js`: AI model management and configuration
- `core/model/StreamingManager.js`: LangChain-powered streaming (exclusive)
- `core/model/LangChainStreamingManager.js`: Core LangChain implementation
- `core/knowledge/KnowledgeManager.js`: RAG and knowledge management
- `core/prompt/PromptBuilder.js`: Dynamic prompt construction

### Tool Modules
- `core/tools/FileTools.js`: File operations (read, write, search)
- `core/tools/SystemTools.js`: System commands and process management
- `core/tools/KnowledgeTools.js`: RAG queries and knowledge operations
- `core/tools/UtilityTools.js`: Utility functions and helpers

### Security Modules
- `core/secure/CommandExecutor.js`: Secure command execution
- `core/secure/CommandRiskAnalyzer.js`: Risk assessment for commands
- `core/secure/AuditLogger.js`: Security event logging
- `core/secure/ConfirmationManager.js`: User confirmation workflows
- `core/secure/WorkspaceManager.js`: Workspace security and management

### Interface Modules
- `interfaces/web-modular.js`: Web interface orchestrator
- `interfaces/cli.js`: Command-line interface
- `interfaces/session/ConversationStore.js`: Conversation persistence
- `interfaces/session/TerminalSessionManager.js`: Terminal session management
- `interfaces/routes/AssistantRoutes.js`: AI assistant API endpoints
- `interfaces/routes/ConversationRoutes.js`: Conversation management API
- `interfaces/routes/ModelRoutes.js`: Model management API
- `interfaces/routes/TerminalRoutes.js`: Terminal API endpoints
- `interfaces/routes/WorkspaceRoutes.js`: Workspace management API

### Services
- `services/scheduler.js`: Task scheduling and background jobs
- `services/openwebui.js`: OpenWebUI integration service

### Utilities
- `utils/logger.js`: Centralized logging utility

## Backend Architecture Diagram (Modular)
```
+-------------------+
|   Entry Point     |
|-------------------|
|     main.js       |
+---------+---------+
          |
          v
+-------------------+
|   Interfaces      |
|-------------------|
| web-modular.js    |
| cli.js            |
| routes/ (modular) |
+---------+---------+
          |
          v
+-------------------+
|  Core Orchestrators|
|-------------------|
| assistant.js      |
| tools.js          |
| secure-executor.js|
+---------+---------+
          |
          v
+-------------------+
| Specialized Mgrs  |
|-------------------|
| ContextManager    |
| ModelManager      |
| StreamingManager  |
| (LangChain-only)  |
| KnowledgeManager  |
+---------+---------+
          |
          v
+-------------------+
|   Tool Modules    |
|-------------------|
| FileTools         |
| SystemTools       |
| KnowledgeTools    |
| UtilityTools      |
+---------+---------+
          |
          v
+-------------------+
| Security Modules  |
|-------------------|
| CommandExecutor   |
| RiskAnalyzer      |
| AuditLogger       |
| ConfirmationMgr   |
+-------------------+
```

## Request Flow (Modular)
1. **Request received** (API, WebSocket, or CLI)
2. Routed to appropriate handler in `interfaces/routes/`
3. Orchestrator delegates to specialized managers (`assistant.js`, `tools.js`)
4. Managers coordinate with specialized modules (Context, Model, Tool modules)
5. If streaming: `StreamingManager` (LangChain-powered) handles response
6. If tool/system command: Security modules handle risk analysis and execution
7. Response returned to client with proper formatting

## Extending the Backend

### Adding New Tools
1. Create tool module in `core/tools/` (e.g., `NewTool.js`)
2. Register in `core/tools.js` orchestrator
3. Add security rules if needed in security modules

### Adding New API Endpoints
1. Create route module in `interfaces/routes/` (e.g., `NewRoutes.js`)
2. Register routes in `interfaces/web-modular.js`
3. Update API documentation

### Adding New Services
1. Create service in `services/` (e.g., `newService.js`)
2. Initialize in `main.js`
3. Configure via environment variables

### Adding New Models
1. Update `core/model/ModelManager.js` configuration
2. Test with `StreamingManager` (LangChain integration)
3. Update environment configuration

## Best Practices
- **Keep modules focused**: Each module should have a single, clear responsibility
- **Use orchestrators**: Don't call modules directly; use orchestrators (`assistant.js`, `tools.js`)
- **Environment configuration**: Use `.env` for all configuration (no hardcoded values)
- **Security first**: All sensitive operations go through security modules
- **Audit everything**: Use `AuditLogger` for all security-relevant actions
- **Test modularly**: Write tests for individual modules (see `test/`)
- **Document changes**: Update wiki and API documentation for new features
- **Use Docker**: Container-based deployment for isolation and scalability

## Configuration Management
- **Environment variables**: All configuration via `.env` file
- **Feature flags**: Enable/disable features (file ops, system commands, RAG)
- **Model configuration**: Dynamic model switching and configuration
- **Security levels**: Configurable risk thresholds and confirmation requirements

## Advanced Usage
- **Custom tools**: Add domain-specific tools for your workflows
- **External APIs**: Integrate via new services or tool modules
- **Custom streaming**: Extend LangChain integration for specialized use cases
- **Multi-tenant**: Workspace isolation and user-specific configurations

## Performance Considerations
- **Modular loading**: Only load required modules based on configuration
- **Streaming efficiency**: LangChain provides optimized streaming
- **Caching**: Knowledge and context caching for improved response times
- **Resource management**: Proper cleanup of terminal sessions and model connections

See the [API Reference](../api/Reference.md) for detailed endpoint documentation and the [Structure Architecture](../structure/Architecture.md) for the full project overview.