drjones/trustos
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: Add comprehensive documentation suite

Browse Source 
- Enhance README.md with full installation instructions, architecture overview, features, configuration, testing, deployment, security, and troubleshooting sections
- Add ARCHITECTURE.md with detailed system architecture, data model, authentication, API design, frontend/backend architecture, database design, AI integration, security, and scalability considerations
-Add API.md with complete API reference including authentication, all endpoints, data models, examples, and interactive documentation links
- Add DEPLOYMENT.md with deployment guides for Railway, Render, VPS, and Kubernetes, including pre-deployment checklist, monitoring, backup, and troubleshooting
- Add CONTRIBUTING.md with development workflow, coding standards, testing guidelines, documentation standards, PR process, and community guidelines
This commit is contained in:
drjones
2026-07-06 04:46:47 +00:00
parent 631a6b4147
commit 9dbf59b995
5 changed files with 4625 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1032
docs/API.md Normal file
View File

File diff suppressed because it is too large Load Diff

980
docs/ARCHITECTURE.md Normal file
View File

@@ -0,0 +1,980 @@
# TrustOS Architecture Documentation
This document provides a detailed overview of the TrustOS system architecture, design decisions, data flow, and technical implementation details.
---
## Table of Contents
- [System Overview](#system-overview)
- [Architecture Principles](#architecture-principles)
- [Component Architecture](#component-architecture)
- [Data Model](#data-model)
- [Authentication & Authorization](#authentication--authorization)
- [API Design](#api-design)
- [Frontend Architecture](#frontend-architecture)
- [Backend Architecture](#backend-architecture)
- [Database Design](#database-design)
- [AI Integration](#ai-integration)
- [Security Architecture](#security-architecture)
- [Scalability Considerations](#scalability-considerations)
- [Performance Optimization](#performance-optimization)
---
## System Overview
TrustOS is a multi-tenant, AI-powered cyber resilience platform built on a modern microservices-inspired architecture. The system is designed to be:
- **Secure**: Multi-tenant isolation with defense-in-depth security
- **Scalable**: Async I/O throughout for high concurrency
- **Maintainable**: Clean separation of concerns and modular design
- **Resilient**: Graceful degradation when external services are unavailable
### High-Level Architecture
```
┌─────────────────────────────────────────────────────────────────┐
│ Client Layer │
│ Web Browser (Executive, IT Admin, TrustOS Admin) │
└────────────────────┬────────────────────────────────────────────┘
│ HTTPS
┌────────────────────▼────────────────────────────────────────────┐
│ Frontend Layer │
│ Next.js 16 + TypeScript + Tailwind CSS + shadcn/ui │
│ - Server-Side Rendering (SSR) │
│ - Client-Side Hydration │
│ - Static Site Generation (SSG) where applicable │
└────────────────────┬────────────────────────────────────────────┘
│ REST API (JSON)
┌────────────────────▼────────────────────────────────────────────┐
│ API Gateway │
│ FastAPI Application │
│ - Request Validation (Pydantic) │
│ - Authentication (JWT) │
│ - Authorization (RBAC) │
│ - Rate Limiting (future) │
│ - Request Logging │
└────────────────────┬────────────────────────────────────────────┘
│
┌────────────┴────────────┐
│ │
┌───────▼────────┐ ┌────────▼─────────┐
│ Service Layer │ │ Background │
│ │ │ Workers │
│ - Dashboard │ │ - AI Translation│
│ - Findings │ │ - Risk Calc │
│ - Reports │ │ - PDF Gen │
│ - Footprint │ │ - Monitoring │
└───────┬────────┘ └────────┬─────────┘
│ │
┌───────▼─────────────────────────▼──────────┐
│ Data Access Layer │
│ SQLAlchemy 2.0 (Async ORM) │
│ - Query Building │
│ - Connection Pooling │
│ - Transaction Management │
└────────────────────┬─────────────────────────┘
│
┌────────────────────▼─────────────────────────┐
│ Database Layer │
│ PostgreSQL 16 │
│ - Multi-Tenant Data Isolation │
│ - Indexing Strategy │
│ - Foreign Key Constraints │
│ - JSONB for Flexible Data │
└──────────────────────────────────────────────┘
External Services:
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ OpenAI API │ │ Anthropic API│ │ HIBP API │
└──────────────┘ └──────────────┘ └──────────────┘
```
---
## Architecture Principles
### 1. Authorization First
TrustOS only scans and monitors assets that have been explicitly authorized by the tenant organization. This is a core security and privacy principle:
- **Authorized Assets Table**: All monitored assets must be pre-registered
- **Scope Enforcement**: All automated checks respect the authorized asset list
- **Executive Enrollment**: Executive monitoring requires explicit organizational consent
- **Audit Trail**: All authorization decisions are logged
### 2. Multi-Tenant Isolation
Every data access path enforces tenant isolation at multiple layers:
- **Database Level**: All tables include `tenant_id` with foreign key constraints
- **ORM Level**: Queries automatically filter by tenant_id
- **API Level**: Middleware validates tenant access before processing requests
- **Application Level**: UI components only display tenant-specific data
### 3. Privacy by Design
Executive and organizational data is handled with privacy as a foundational requirement:
- **Minimal Data Collection**: Only collect data necessary for security assessments
- **Explicit Consent**: Executive enrollment requires organizational authorization
- **Data Minimization**: Store only what, not who, where possible
- **Audit Logging**: All data access is logged for accountability
### 4. AI-Augmented, Not AI-Dependent
AI features enhance the product but are not required for core functionality:
- **Graceful Degradation**: Features work with raw technical data if AI is unavailable
- **Caching**: AI responses are cached to avoid redundant API calls
- **Fallback Mechanisms**: System continues operating if AI services are down
- **Cost Control**: Rate limiting and caching to manage AI API costs
### 5. Audit Trail
All state changes are tracked with full provenance:
- **Timestamps**: `created_at` and `updated_at` on all records
- **User Attribution**: `created_by` and `updated_by` where applicable
- **Status Changes**: Finding status transitions are logged with notes
- **Access Logs**: API requests are logged with user and tenant context
---
## Component Architecture
### Frontend Components
#### Page Components
- **Dashboard Page** (`/dashboard`): Executive view with risk score, top risks, trends
- **Findings Page** (`/findings`): IT admin view with sortable/filterable table
- **Finding Detail Page** (`/findings/[id]`): Detailed view with technical and business impact
- **Login Page** (`/login`): Authentication interface
- **Footprint Page** (`/footprint`): Digital footprint center
#### Reusable Components)
- **RiskDial**: Circular gauge for cyber health score (0-100)
- **RiskCard**: Card displaying finding with AI summary and impact
- **TrendChart**: Line chart for 90-day risk history
- **RemediationBoard**: Kanban board for finding status tracking
- **Badge**: Severity and status badges with color coding
#### Hooks
- **useAuth**: Authentication state management (token, role, tenant_id)
- **useDashboard**: Dashboard data fetching and caching
- **useFindings**: Findings list and detail fetching
### Backend Components
#### API Routes
- **auth.py**: Login, token refresh, user info
- **dashboard.py**: Dashboard data aggregation
- **findings.py**: Findings CRUD operations
- **reports.py**: Audit report generation
- **attack_paths.py**: Attack path visualization
- **footprint.py**: Digital footprint data
- **ai.py**: AI translation and coaching
#### Services
- **ai_translator.py**: OpenAI/Anthropic integration for risk translation
- **risk_calculator.py**: Risk score calculation algorithm
- **report_generator.py**: PDF report generation with Jinja2/WeasyPrint
#### Core
- **config.py**: Configuration management with Pydantic Settings
- **security.py**: JWT token management, password hashing, RBAC decorators
---
## Data Model
### Entity Relationship Diagram
```
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ tenants │───────│ users │───────│ findings │
│─────────────│ 1:N │─────────────│ 1:N │─────────────│
│ id (PK) │ │ id (PK) │ │ id (PK) │
│ name │ │ tenant_id │ │ tenant_id │
│ slug │ │ email │ │ asset_id │
│ industry │ │ role │ │ executive_id│
│ size_range │ │ ... │ │ severity │
│ ... │ └─────────────┘ │ status │
└─────────────┘ │ category │
│ │ ai_summary │
│ │ ... │
│ └─────────────┘
│ │
│ │
┌─────────────┐ ┌───────────▼──────────┐
│ assets │ │ risk_scores │
│─────────────│ │──────────────────────│
│ id (PK) │ │ id (PK) │
│ tenant_id │ │ tenant_id │
│ name │ │ score_date │
│ asset_type │ │ overall_score │
│ value │ │ score_identity │
│ ... │ │ score_cloud │
└─────────────┘ │ ... │
└──────────────────────┘
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ executives │ │authorized │ │attack_paths │
│─────────────│ │ assets │ │─────────────│
│ id (PK) │ │─────────────│ │ id (PK) │
│ tenant_id │ │ id (PK) │ │ finding_id │
│ full_name │ │ tenant_id │ │ title │
│ title │ │ value │ │ ai_narrative│
│ email │ │ asset_type │ │ nodes_json │
│ ... │ │ ... │ │ edges_json │
└─────────────┘ └─────────────┘ └─────────────┘
┌─────────────┐
│audit_reports│
│─────────────│
│ id (PK) │
│ tenant_id │
│ title │
│ report_date │
│ baseline_ │
│ score │
│ pdf_path │
│ ... │
└─────────────┘
```
### Key Tables
#### tenants
Organizational units with complete data isolation.
- `id`: UUID primary key
- `name`: Organization display name
- `slug`: URL-friendly identifier (unique)
- `industry`: Industry classification
- `size_range`: Company size (SMB, mid-market, enterprise)
- `contact_email`: Primary contact
- `is_active`: Active status for soft deletes
#### users
User accounts with role-based access control.
- `id`: UUID primary key
- `tenant_id`: Foreign key to tenants
- `email`: Unique email address
- `hashed_password`: Bcrypt hash
- `role`: enum (executive, it_admin, trustos_admin)
- `is_active`: Account status
#### findings
Security vulnerabilities and exposures.
- `id`: UUID primary key
- `tenant_id`: Foreign key to tenants
- `asset_id`: Foreign key to assets (nullable)
- `executive_id`: Foreign key to executives (nullable)
- `title`: Human-readable title
- `severity`: enum (critical, high, medium, low, info)
- `status`: enum (open, in_progress, resolved, verified)
- `category`: enum (external_exposure, cloud_posture, credential_exposure, etc.)
- `technical_description`: Raw technical details
- `cve_id`: CVE identifier (if applicable)
- `cvss_score`: CVSS score (if applicable)
- `ai_summary`: AI-generated plain-English summary
- `ai_business_impact`: AI-generated business impact
- `ai_remediation_steps`: AI-generated fix steps
- `assignee_email`: Assigned team member
- `due_date`: Remediation deadline
- `is_top_risk`: Flag for top 3 risks
#### risk_scores
Daily snapshots of risk metrics.
- `id`: UUID primary key
- `tenant_id`: Foreign key to tenants
- `score_date`: Timestamp of snapshot
- `overall_score`: 0-100 overall score
- `score_identity`: Identity security score
- `score_cloud`: Cloud posture score
- `score_network`: Network security score
- `score_web`: Web application score
- `score_credential`: Credential security score
- `score_digital_footprint`: Digital footprint score
- `score_third_party`: Third-party risk score
- `critical_count`: Count of critical findings
- `high_count`: Count of high findings
- `medium_count`: Count of medium findings
- `low_count`: Count of low findings
---
## Authentication & Authorization
### Authentication Flow
```
1. User submits credentials to POST /api/v1/auth/login
↓
2. Backend validates credentials against database
↓
3. Backend generates JWT token with:
- sub: user_id
- role: user_role
- tenant_id: tenant_id
- exp: expiration timestamp
↓
4. Frontend stores token in localStorage
↓
5. Frontend includes token in Authorization header: Bearer <token>
↓
6. Backend validates token on each protected request
↓
7. Backend extracts user context from token
↓
8. Request proceeds with user context
```
### JWT Token Structure
```json
{
"sub": "user-uuid",
"role": "executive",
"tenant_id": "tenant-uuid",
"exp": 1234567890,
"iat": 1234567890
}
```
### Role-Based Access Control (RBAC)
Three roles with distinct permissions:
#### Executive
- **Can View**: Dashboard, risk scores, AI summaries, trends
- **Cannot View**: Raw CVE data, technical details, other tenants
- **Can Modify**: None (read-only)
#### IT Admin
- **Can View**: All findings, technical details, CVEs, remediation steps
- **Can Modify**: Finding status, assignees, due dates, resolution notes
- **Cannot View**: Other tenants' data
#### TrustOS Admin
- **Can View**: All tenants, all data, system metrics
- **Can Modify**: Tenant settings, authorized assets, audit reports
- **Can Manage**: Users, roles, system configuration
### Authorization Middleware
```python
# Example from app/core/security.py
def require_executive_or_above(payload: dict = Depends(verify_token)):
if payload.get("role") not in ["executive", "it_admin", "trustos_admin"]:
raise HTTPException(status_code=403, detail="Insufficient permissions")
return payload
def require_it_or_above(payload: dict = Depends(verify_token)):
if payload.get("role") not in ["it_admin", "trustos_admin"]:
raise HTTPException(status_code=403, detail="Insufficient permissions")
return payload
def require_admin(payload: dict = Depends(verify_token)):
if payload.get("role") != "trustos_admin":
raise HTTPException(status_code=403, detail="Admin access required")
return payload
```
---
## API Design
### RESTful Conventions
TrustOS follows RESTful API design principles:
- **Resource-Based URLs**: `/api/v1/findings`, `/api/v1/tenants`
- **HTTP Methods**: GET (read), POST (create), PATCH (update), DELETE (delete)
- **Status Codes**: 200 (success), 201 (created), 400 (bad request), 401 (unauthorized), 403 (forbidden), 404 (not found), 500 (server error)
- **JSON Request/Response**: All data is JSON-encoded
- **Versioning**: `/api/v1/` prefix for future compatibility
### API Response Format
#### Success Response
```json
{
"data": { ... },
"meta": {
"timestamp": "2024-01-01T00:00:00Z",
"request_id": "uuid"
}
}
```
#### Error Response
```json
{
"error": {
"code": "VALIDATION_ERROR",
"message": "Invalid input data",
"details": { ... }
},
"meta": {
"timestamp": "2024-01-01T00:00:00Z",
"request_id": "uuid"
}
}
```
### Key API Endpoints
#### Authentication
- `POST /api/v1/auth/login` - Authenticate and receive token
- `GET /api/v1/auth/me` - Get current user info
#### Dashboard
- `GET /api/v1/dashboard?tenant_id={id}` - Get dashboard data
#### Findings
- `GET /api/v1/findings?tenant_id={id}` - List findings
- `GET /api/v1/findings/{id}` - Get finding detail
- `POST /api/v1/findings` - Create finding (IT Admin+)
- `PATCH /api/v1/findings/{id}/status` - Update finding status (IT Admin+)
#### Reports
- `GET /api/v1/audit-reports?tenant_id={id}` - List audit reports
- `POST /api/v1/audit-reports/generate` - Generate audit report (Admin)
#### AI
- `POST /api/v1/ai/translate/{finding_id}` - Trigger AI translation (IT Admin+)
- `GET /api/v1/ai/explain/{finding_id}?question=...` - AI Security Coach (IT Admin+)
---
## Frontend Architecture
### Next.js App Router Structure
```
src/app/
├── layout.tsx # Root layout with providers
├── page.tsx # Root redirect to /login
├── login/
│ └── page.tsx # Login page
├── dashboard/
│ └── page.tsx # Executive dashboard
├── findings/
│ ├── page.tsx # Findings list
│ └── [id]/
│ └── page.tsx # Finding detail
└── footprint/
└── page.tsx # Digital footprint center
```
### State Management
TrustOS uses React Context for global state:
```typescript
// Auth Context
interface AuthContext {
token: string | null;
role: UserRole | null;
tenantId: string | null;
name: string | null;
login: (email: string, password: string) => Promise<void>;
logout: () => void;
}
```
### API Client
Custom API client with token management:
```typescript
const api = {
login: (email: string, password: string) =>
fetch('/api/v1/auth/login', { method: 'POST', body: ... }),
dashboard: (tenantId: string) =>
fetch(`/api/v1/dashboard?tenant_id=${tenantId}`, {
headers: { Authorization: `Bearer ${token}` }
}),
// ... other methods
};
```
### Component Design Patterns
#### Presentational Components
- Receive data via props
- No side effects
- Reusable across contexts
#### Container Components
- Fetch data from API
- Manage state
- Pass data to presentational components
#### Higher-Order Components
- `withAuth`: Wraps components requiring authentication
- `withRole`: Wraps components requiring specific roles
---
## Backend Architecture
### FastAPI Application Structure
```python
# app/main.py
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
app = FastAPI(
title="TrustOS API",
version="1.0.0",
description="AI-powered cyber resilience platform"
)
# CORS middleware
app.add_middleware(
CORSMiddleware,
allow_origins=["http://localhost:3000"],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
# Include routers
app.include_router(auth.router, prefix="/api/v1/auth", tags=["auth"])
app.include_router(dashboard.router, prefix="/api/v1/dashboard", tags=["dashboard"])
# ... other routers
@app.on_event("startup")
async def startup_event():
# Create tables in development
if settings.DEBUG:
async with engine.begin() as conn:
await conn.run_sync(Base.metadata.create_all)
@app.get("/health")
async def health_check():
return {"status": "healthy"}
```
### Dependency Injection
FastAPI's dependency system for authentication and database:
```python
# Database session
async def get_db() -> AsyncSession:
async with AsyncSessionLocal() as session:
try:
yield session
finally:
await session.close()
# Authentication
async def verify_token(authorization: str = Header(...)) -> dict:
token = authorization.replace("Bearer ", "")
payload = decode_jwt(token)
return payload
# Protected route
@router.get("/dashboard")
async def get_dashboard(
tenant_id: str,
payload: dict = Depends(verify_token),
db: AsyncSession = Depends(get_db)
):
# Access payload['role'], payload['tenant_id']
# Use db for database operations
...
```
### Service Layer Pattern
Business logic separated from API routes:
```python
# app/services/risk_calculator.py
async def recalculate_risk_score(tenant_id: str, db: AsyncSession) -> float:
# Business logic for risk calculation
findings = await get_open_findings(tenant_id, db)
score = calculate_score(findings)
await save_risk_snapshot(tenant_id, score, db)
return score
# API route uses service
@router.patch("/findings/{id}/status")
async def update_status(finding_id: str, body: StatusUpdate, db: AsyncSession = Depends(get_db)):
finding = await get_finding(finding_id, db)
finding.status = body.status
await db.commit()
# Trigger background risk recalculation
asyncio.create_task(recalculate_risk_score(finding.tenant_id))
return finding
```
---
## Database Design
### Schema Design Principles
1. **Multi-Tenant by Default**: All tables include `tenant_id`
2. **UUID Primary Keys**: Distributed-friendly, no sequence contention
3. **Soft Deletes**: `is_active` flags instead of hard deletes
4. **Audit Fields**: `created_at`, `updated_at` on all tables
5. **JSONB for Flexibility**: Store semi-structured data in JSONB columns
### Indexing Strategy
```sql
-- Tenant isolation (every query)
CREATE INDEX idx_findings_tenant_id ON findings(tenant_id);
-- Common query patterns
CREATE INDEX idx_findings_status ON findings(status);
CREATE INDEX idx_findings_severity ON findings(severity);
CREATE INDEX idx_findings_category ON findings(category);
CREATE INDEX idx_findings_tenant_status ON findings(tenant_id, status);
-- Time-series queries
CREATE INDEX idx_risk_scores_tenant_date ON risk_scores(tenant_id, score_date DESC);
-- Unique constraints
CREATE UNIQUE INDEX idx_users_email ON users(email);
CREATE UNIQUE INDEX idx_tenants_slug ON tenants(slug);
```
### Connection Pooling
```python
# SQLAlchemy async engine with connection pooling
engine = create_async_engine(
settings.DATABASE_URL,
echo=False,
pool_pre_ping=True, # Verify connections before use
pool_size=10, # Base pool size
max_overflow=20, # Additional connections under load
)
```
---
## AI Integration
### AI Service Architecture
```
┌─────────────┐
│ API Route │
└──────┬──────┘
│
┌──────▼──────────┐
│ AI Translator │
│ Service │
└──────┬──────────┘
│
┌──────▼──────────┐
│ LLM Provider │
│ (OpenAI/Anthropic)│
└─────────────────┘
```
### AI Translation Flow
```python
# 1. Finding created or updated
finding = Finding(...)
# 2. Trigger AI translation (background task)
asyncio.create_task(translate_finding_async(finding.id))
# 3. AI service constructs prompt
prompt = f"""
Translate this cybersecurity finding:
Title: {finding.title}
Severity: {finding.severity}
CVE ID: {finding.cve_id}
Technical Description: {finding.technical_description}
"""
# 4. Call LLM with system prompt
system_prompt = """
You are TrustOS, an AI cyber resilience advisor.
Translate technical findings into plain-English business impact.
Output JSON with: summary, business_impact, impact_level, remediation_steps.
"""
# 5. Parse and store AI response
data = json.loads(llm_response)
finding.ai_summary = data["summary"]
finding.ai_business_impact = data["business_impact"]
finding.ai_remediation_steps = data["remediation_steps"]
await db.commit()
```
### AI Provider Selection
```python
# app/core/config.py
AI_PROVIDER = os.getenv("AI_PROVIDER", "openai") # or "anthropic"
# app/services/ai_translator.py
if settings.AI_PROVIDER == "openai":
client = AsyncOpenAI(api_key=settings.OPENAI_API_KEY)
response = await client.chat.completions.create(
model="gpt-4o-mini",
messages=[...],
response_format={"type": "json_object"}
)
elif settings.AI_PROVIDER == "anthropic":
client = AsyncAnthropic(api_key=settings.ANTHROPIC_API_KEY)
response = await client.messages.create(
model="claude-3-haiku-20240307",
messages=[...]
)
```
### AI Caching Strategy
- **Store AI Responses**: AI translations are stored in the database
- **Avoid Redundant Calls**: Check if `ai_summary` exists before re-translating
- **Background Processing**: AI calls are async and non-blocking
- **Graceful Degradation**: If AI fails, use raw technical description
---
## Security Architecture
### Defense in Depth
```
┌─────────────────────────────────────────────────────────┐
│ 1. Network Security │
│ - HTTPS/TLS encryption │
│ - CORS restrictions │
│ - Rate limiting (planned) │
└─────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────┐
│ 2. Authentication │
│ - JWT token validation │
│ - Token expiration (8 hours) │
│ - Secure password hashing (bcrypt) │
└─────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────┐
│ 3. Authorization │
│ - Role-based access control │
│ - Tenant isolation enforcement │
│ - Route-level permission checks │
└─────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────┐
│ 4. Input Validation │
│ - Pydantic schema validation │
│ - SQL injection prevention (ORM) │
│ - XSS prevention (React escaping) │
└─────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────┐
│ 5. Data Security │
│ - Multi-tenant database isolation │
│ - Encrypted secrets management │
│ - Audit logging │
└─────────────────────────────────────────────────────────┘
```
### Security Headers
```python
# app/main.py
from fastapi.middleware.trustedhost import TrustedHostMiddleware
app.add_middleware(
TrustedHostMiddleware,
allowed_hosts=["trustos.com", "*.trustos.com"]
)
# Additional headers via middleware
@app.middleware("http")
async def add_security_headers(request: Request, call_next):
response = await call_next(request)
response.headers["X-Content-Type-Options"] = "nosniff"
response.headers["X-Frame-Options"] = "DENY"
response.headers["X-XSS-Protection"] = "1; mode=block"
response.headers["Strict-Transport-Security"] = "max-age=31536000; includeSubDomains"
return response
```
### Secrets Management
- **Environment Variables**: All secrets in `.env` files (git-ignored)
- **Production**: Use secret management services (AWS Secrets Manager, HashiCorp Vault)
- **Rotation**: Regular API key rotation policy
- **Least Privilege**: Database users have minimal required permissions
---
## Scalability Considerations
### Horizontal Scaling
The architecture supports horizontal scaling:
- **Stateless API**: FastAPI instances can be scaled horizontally
- **Database Connection Pooling**: Efficient connection reuse
- **Async I/O**: High concurrency with minimal threads
- **Load Balancer**: Nginx or cloud load balancer in front of API
### Vertical Scaling
- **Database**: PostgreSQL can scale vertically (more CPU/RAM)
- **Caching**: Redis planned for session and query caching
- **CDN**: Frontend static assets served via CDN
### Performance Optimization Strategies
1. **Database Optimization**
- Proper indexing on frequently queried columns
- Query optimization with EXPLAIN ANALYZE
- Connection pooling to reduce overhead
- Read replicas for read-heavy workloads (future)
2. **API Optimization**
- Response compression (gzip)
- Pagination for large result sets
- Selective field loading (avoid SELECT *)
- Async operations throughout
3. **Frontend Optimization**
- Code splitting with Next.js
- Image optimization
- Static generation where possible
- Client-side caching
4. **Caching Strategy**
- API response caching (Redis)
- Static asset caching (CDN)
- AI response caching (database)
- Browser caching headers
---
## Performance Optimization
### Database Query Optimization
```python
# Bad: N+1 query problem
for finding in findings:
asset = await get_asset(finding.asset_id) # N queries
# Good: Eager loading
findings = await db.execute(
select(Finding).options(selectinload(Finding.asset))
)
```
### API Response Optimization
```python
# Bad: Return all fields
@router.get("/findings")
async def list_findings():
return await db.execute(select(Finding))
# Good: Select only needed fields
@router.get("/findings")
async def list_findings():
return await db.execute(
select(Finding.id, Finding.title, Finding.severity)
)
```
### Frontend Performance
```typescript
// Bad: Re-render on every state change
useEffect(() => {
fetchData();
}, [state]); // Runs on every state change
// Good: Only re-fetch when dependencies change
useEffect(() => {
fetchData();
}, [tenantId, filter]); // Only runs when tenant or filter changes
```
---
## Monitoring & Observability
### Logging Strategy
```python
# Structured logging
import logging
logger = logging.getLogger(__name__)
logger.info(
"Finding status updated",
extra={
"finding_id": finding.id,
"old_status": old_status,
"new_status": new_status,
"user_id": user_id,
"tenant_id": tenant_id
}
)
```
### Metrics to Track
- **API Metrics**: Request rate, error rate, response time
- **Database Metrics**: Query time, connection pool usage
- **Business Metrics**: Active tenants, findings created, risk score trends
- **AI Metrics**: API calls, token usage, cost tracking
### Health Checks
```python
@app.get("/health")
async def health_check():
checks = {
"database": await check_database(),
"ai_service": await check_ai_service(),
"storage": await check_storage()
}
status = "healthy" if all(checks.values()) else "degraded"
return {"status": status, "checks": checks}
```
---
## Future Architecture Enhancements
### Planned Improvements
1. **Message Queue**: Celery + Redis for background job processing
2. **Caching Layer**: Redis for session and query caching
3. **Read Replicas**: PostgreSQL read replicas for scaling reads
4. **Microservices**: Split into separate services (auth, findings, monitoring)
5. **Event Sourcing**: Event-driven architecture for audit trail
6. **GraphQL**: Alternative to REST for complex queries
7. **Real-time Updates**: WebSocket for live dashboard updates
8. **Edge Computing**: Cloudflare Workers for global distribution
---
## Conclusion
The TrustOS architecture is designed to be secure, scalable, and maintainable while following modern best practices. The multi-tenant isolation, AI augmentation, and authorization-first principles ensure the platform can grow with customer needs while maintaining security and privacy.
For questions or contributions to the architecture, please refer to the [CONTRIBUTING.md](CONTRIBUTING.md) guide.

768
docs/CONTRIBUTING.md Normal file
View File

@@ -0,0 +1,768 @@
# Contributing to TrustOS
Thank you for your interest in contributing to TrustOS! This document provides guidelines and instructions for contributing to the project.
---
## Table of Contents
- [Code of Conduct](#code-of-conduct)
- [Getting Started](#getting-started)
- [Development Workflow](#development-workflow)
- [Coding Standards](#coding-standards)
- [Testing Guidelines](#testing-guidelines)
- [Documentation Guidelines](#documentation-guidelines)
- [Pull Request Process](#pull-request-process)
- [Commit Message Guidelines](#commit-message-guidelines)
- [Reporting Issues](#reporting-issues)
- [Feature Requests](#feature-requests)
- [Community Guidelines](#community-guidelines)
---
## Code of Conduct
### Our Pledge
We are committed to providing a welcoming and inclusive environment for all contributors. We value diverse perspectives and constructive collaboration.
### Our Standards
- Be respectful and considerate
- Use inclusive language
- Focus on constructive feedback
- Accept feedback gracefully
- Show empathy toward other community members
### Unacceptable Behavior
- Harassment or discrimination
- Personal attacks
- Derogatory comments
- Public or private harassment
- Publishing others' private information
### Reporting Issues
If you witness or experience unacceptable behavior, please contact the project maintainers at conduct@trustos.com.
---
## Getting Started
### Prerequisites
Before contributing, ensure you have:
- **Git** installed and configured
- **Python 3.10+** for backend development
- **Node.js 18+** and **npm 9+** for frontend development
- **Docker** and **Docker Compose** for local development
- Basic knowledge of Python, TypeScript, React, and FastAPI
### Setting Up Development Environment
1. **Fork the Repository**
```bash
# Fork the repository on GitHub/Gitea
# Clone your fork
git clone https://gitea.thetempleofdoom.com/YOUR_USERNAME/trustos.git
cd trustos
```
2. **Set Up Backend**
```bash
cd backend
python3 -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
pip install -r requirements.txt
cp .env.example .env
# Edit .env with your configuration
```
3. **Set Up Frontend**
```bash
cd frontend
npm install
cp .env.example .env.local
# Edit .env.local with NEXT_PUBLIC_API_URL=http://localhost:8000
```
4. **Start Development Services**
```bash
# From project root
cd infra
docker-compose up -d postgres
```
5. **Initialize Database**
```bash
cd backend
python seed.py
```
6. **Run Development Servers**
```bash
# Terminal 1: Backend
cd backend
uvicorn app.main:app --reload --host 0.0.0.0 --port 8000
# Terminal 2: Frontend
cd frontend
npm run dev
```
### Development Tools
#### Backend Tools
- **Black**: Code formatting
```bash
pip install black
black app/
```
- **isort**: Import sorting
```bash
pip install isort
isort app/
```
- **flake8**: Linting
```bash
pip install flake8
flake8 app/
```
- **mypy**: Type checking
```bash
pip install mypy
mypy app/
```
#### Frontend Tools
- **ESLint**: Linting
```bash
npm run lint
```
- **Prettier**: Code formatting
```bash
npm run format
```
- **TypeScript**: Type checking
```bash
npm run type-check
```
---
## Development Workflow
### 1. Create a Branch
Create a new branch for your work:
```bash
git checkout -b feature/your-feature-name
# or
git checkout -b fix/your-bug-fix
```
Branch naming conventions:
- `feature/` - New features
- `fix/` - Bug fixes
- `docs/` - Documentation changes
- `refactor/` - Code refactoring
- `test/` - Test additions/changes
- `chore/` - Maintenance tasks
### 2. Make Your Changes
- Write clean, readable code
- Follow coding standards (see below)
- Add tests for new functionality
- Update documentation as needed
### 3. Test Your Changes
```bash
# Backend tests
cd backend
pytest
# Frontend tests
cd frontend
npm test
# Manual testing
# - Test the feature in the browser
# - Test API endpoints at /docs
# - Verify no regressions
```
### 4. Commit Your Changes
Follow commit message guidelines (see below):
```bash
git add .
git commit -m "feat: add user authentication"
```
### 5. Push to Your Fork
```bash
git push origin feature/your-feature-name
```
### 6. Create a Pull Request
- Go to the repository on GitHub/Gitea
- Click "New Pull Request"
- Provide a clear description of your changes
- Link to any related issues
- Request review from maintainers
---
## Coding Standards
### Python (Backend)
#### Style Guide
Follow [PEP 8](https://pep8.org/) style guidelines:
```python
# Good
def calculate_risk_score(findings: List[Finding]) -> float:
"""Calculate the overall risk score from a list of findings."""
score = 100.0
for finding in findings:
score -= SEVERITY_DEDUCTIONS[finding.severity]
return max(0.0, score)
# Bad
def calc(f):
s=100
for x in f:
s-=d[x.sev]
return max(0,s)
```
#### Type Hints
Use type hints for all function signatures:
```python
from typing import List, Optional
def get_findings(
tenant_id: str,
severity: Optional[str] = None,
limit: int = 50
) -> List[Finding]:
...
```
#### Docstrings
Use Google-style docstrings:
```python
def calculate_risk_score(findings: List[Finding]) -> float:
"""Calculate the overall risk score from a list of findings.
Args:
findings: List of findings to calculate score from.
Returns:
Overall risk score between 0 and 100.
Raises:
ValueError: If findings list is empty.
"""
...
```
#### Error Handling
Use specific exceptions:
```python
# Good
try:
finding = await get_finding(finding_id)
except NotFoundError:
raise HTTPException(status_code=404, detail="Finding not found")
# Bad
try:
finding = await get_finding(finding_id)
except:
raise HTTPException(status_code=500)
```
### TypeScript (Frontend)
#### Style Guide
Follow ESLint rules and Prettier configuration:
```typescript
// Good
interface DashboardData {
currentScore: number;
topRisks: RiskCard[];
scoreTrend: TrendPoint[];
}
async function fetchDashboard(tenantId: string): Promise<DashboardData> {
const response = await api.dashboard(tenantId);
return response.data;
}
// Bad
interface d {
s: number;
r: any[];
}
async function f(id: string) {
const r = await api.dashboard(id);
return r.data;
}
```
#### Component Structure
```typescript
// Good: Clear component structure
interface RiskCardProps {
finding: Finding;
onDetailClick: (id: string) => void;
}
export function RiskCard({ finding, onDetailClick }: RiskCardProps) {
const [isExpanded, setIsExpanded] = useState(false);
return (
<div className="vault-card">
{/* Component content */}
</div>
);
}
```
#### Hooks
Use custom hooks for reusable logic:
```typescript
// Good: Custom hook for data fetching
function useFindings(tenantId: string) {
const [findings, setFindings] = useState<Finding[]>([]);
const [loading, setLoading] = useState(true);
useEffect(() => {
fetchFindings(tenantId).then(data => {
setFindings(data);
setLoading(false);
});
}, [tenantId]);
return { findings, loading };
}
```
### General Guidelines
- **Keep functions small**: Functions should do one thing well
- **Avoid deep nesting**: Use early returns to reduce nesting
- **Use meaningful names**: Variable and function names should be descriptive
- **Comment complex logic**: Explain why, not what
- **DRY principle**: Don't repeat yourself
- **YAGNI principle**: You aren't gonna need it (avoid over-engineering)
---
## Testing Guidelines
### Backend Testing
#### Test Structure
```python
# tests/test_findings.py
import pytest
from app.models.models import Finding, FindingSeverity
class TestFindings:
"""Test suite for findings endpoints."""
@pytest.fixture
async def sample_finding(self, db):
"""Create a sample finding for testing."""
finding = Finding(
tenant_id="test-tenant",
title="Test Finding",
severity=FindingSeverity.high,
status=FindingStatus.open
)
db.add(finding)
await db.commit()
await db.refresh(finding)
return finding
async def test_get_finding(self, client, sample_finding):
"""Test retrieving a single finding."""
response = await client.get(f"/api/v1/findings/{sample_finding.id}")
assert response.status_code == 200
data = response.json()
assert data["id"] == sample_finding.id
```
#### Test Coverage
Aim for at least 80% test coverage:
```bash
# Run with coverage
pytest --cov=app --cov-report=html --cov-report=term
```
#### Testing Best Practices
- Write tests before code (TDD when possible)
- Test both happy path and error cases
- Use fixtures for common test data
- Mock external dependencies (API calls, database)
- Keep tests independent and fast
### Frontend Testing
#### Component Testing
```typescript
// components/__tests__/RiskCard.test.tsx
import { render, screen } from '@testing-library/react';
import { RiskCard } from '../RiskCard';
describe('RiskCard', () => {
it('renders finding title', () => {
const finding = {
id: '1',
title: 'Test Finding',
severity: 'high',
ai_summary: 'Test summary'
};
render(<RiskCard finding={finding} onDetailClick={jest.fn()} />);
expect(screen.getByText('Test Finding')).toBeInTheDocument();
});
});
```
#### Integration Testing
```typescript
// __tests__/dashboard.test.tsx
import { render, screen, waitFor } from '@testing-library/react';
import { Dashboard } from '../app/dashboard/page';
describe('Dashboard', () => {
it('displays risk score after loading', async () => {
render(<Dashboard />);
await waitFor(() => {
expect(screen.getByText(/risk score/i)).toBeInTheDocument();
});
});
});
```
#### Testing Best Practices
- Test user behavior, not implementation details
- Use meaningful test descriptions
- Mock API calls in component tests
- Test accessibility (ARIA labels, keyboard navigation)
- Keep tests focused and independent
---
## Documentation Guidelines
### Code Documentation
- Document all public functions and classes
- Use docstrings for complex logic
- Comment non-obvious algorithms
- Keep documentation up to date with code changes
### API Documentation
- Update OpenAPI/Swagger annotations for new endpoints
- Include request/response examples
- Document error responses
- Note authentication requirements
### README Updates
- Update README.md for user-facing changes
- Add new features to the features list
- Update installation instructions if needed
- Add new environment variables to the configuration section
### Changelog
Maintain a CHANGELOG.md:
```markdown
## [Unreleased]
### Added
- New feature description
### Changed
- Changed behavior description
### Fixed
- Bug fix description
### Deprecated
- Deprecated feature description
```
---
## Pull Request Process
### Before Submitting
- [ ] Code follows project style guidelines
- [ ] Tests pass locally
- [ ] New tests added for new functionality
- [ ] Documentation updated
- [ ] Commit messages follow guidelines
- [ ] No merge conflicts with target branch
### Pull Request Description
Provide a clear description:
```markdown
## Description
Brief description of what this PR does.
## Type of Change
- [ ] Bug fix
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
## Testing
How did you test this change?
## Checklist
- [ ] Tests added/updated
- [ ] Documentation updated
- [ ] No breaking changes (or documented)
```
### Review Process
1. Automated checks (CI/CD) must pass
2. At least one maintainer approval required
3. Address all review comments
4. Update PR as needed
5. Maintainer merges when approved
---
## Commit Message Guidelines
Follow [Conventional Commits](https://www.conventionalcommits.org/) specification:
### Format
```
<type>(<scope>): <subject>
<body>
<footer>
```
### Types
- `feat`: New feature
- `fix`: Bug fix
- `docs`: Documentation changes
- `style`: Code style changes (formatting, etc.)
- `refactor`: Code refactoring
- `test`: Test additions/changes
- `chore`: Maintenance tasks
- `perf`: Performance improvements
- `ci`: CI/CD changes
### Examples
```bash
feat(auth): add OAuth2 authentication support
Implement OAuth2 authentication with Google and GitHub providers.
Users can now sign in using their existing accounts.
Closes #123
fix(dashboard): correct risk score calculation
The risk score calculation was not properly weighting critical findings.
This fix updates the severity deduction algorithm.
Closes #456
docs(readme): update installation instructions
Added Docker Compose setup instructions and updated prerequisites.
docs(api): add authentication endpoint documentation
Documented the new login endpoint with request/response examples.
```
### Guidelines
- Use the imperative mood ("add" not "added")
- Limit first line to 50 characters
- Wrap body at 72 characters
- Reference issue numbers in footer
---
## Reporting Issues
### Bug Reports
When reporting a bug, include:
1. **Description**: Clear description of the bug
2. **Steps to Reproduce**: Detailed steps to reproduce the issue
3. **Expected Behavior**: What you expected to happen
4. **Actual Behavior**: What actually happened
5. **Environment**:
- OS and version
- Python/Node version
- Browser version (if applicable)
6. **Screenshots**: If applicable
7. **Additional Context**: Any other relevant information
### Issue Template
```markdown
## Description
[Clear description of the bug]
## Steps to Reproduce
1. Go to '...'
2. Click on '...'
3. Scroll down to '...'
4. See error
## Expected Behavior
[What you expected to happen]
## Actual Behavior
[What actually happened]
## Environment
- OS: [e.g. Ubuntu 22.04]
- Python: [e.g. 3.10]
- Node: [e.g. 18.0]
- Browser: [e.g. Chrome 120]
## Screenshots
[If applicable]
## Additional Context
[Any other relevant information]
```
---
## Feature Requests
### Proposing a Feature
Before proposing a new feature:
1. Check if the feature already exists
2. Search existing issues to avoid duplicates
3. Consider if the feature aligns with project goals
4. Think about implementation complexity
### Feature Request Template
```markdown
## Feature Description
[Clear description of the feature]
## Problem Statement
[What problem does this feature solve?]
## Proposed Solution
[How should this feature work?]
## Alternatives Considered
[What alternatives did you consider?]
## Additional Context
[Any other relevant information, screenshots, etc.]
```
---
## Community Guidelines
### Communication Channels
- **GitHub/Gitea Issues**: For bug reports and feature requests
- **Discussions**: For questions and general discussion
- **Email**: support@trustos.com for private matters
### Getting Help
1. Check existing documentation
2. Search existing issues
3. Ask in discussions
4. Contact maintainers directly if needed
### Recognition
Contributors will be recognized in:
- CONTRIBUTORS.md file
- Release notes
- Project website (if applicable)
---
## License
By contributing to TrustOS, you agree that your contributions will be licensed under the project's license (Proprietary).
---
## Questions?
If you have questions about contributing, please:
- Open a discussion on GitHub/Gitea
- Email the maintainers at dev@trustos.com
- Check the existing documentation
Thank you for contributing to TrustOS!

1033
docs/DEPLOYMENT.md Normal file
View File

File diff suppressed because it is too large Load Diff