trustos/docs/DEPLOYMENT.md

# TrustOS Deployment Guide

This guide provides comprehensive instructions for deploying TrustOS to production environments.

---

## Table of Contents

- [Overview](#overview)
- [Deployment Options](#deployment-options)
- [Pre-Deployment Checklist](#pre-deployment-checklist)
- [Option 1: Railway Deployment](#option-1-railway-deployment)
- [Option 2: Render Deployment](#option-2-render-deployment)
- [Option 3: VPS Deployment](#option-3-vps-deployment)
- [Option 4: Kubernetes Deployment](#option-4-kubernetes-deployment)
- [Post-Deployment Steps](#post-deployment-steps)
- [Monitoring & Maintenance](#monitoring--maintenance)
- [Backup & Recovery](#backup--recovery)
- [Troubleshooting](#troubleshooting)

---

## Overview

TrustOS can be deployed to various platforms depending on your needs and expertise:

- **Railway**: Easiest option, fully managed, good for quick deployment
- **Render**: Simple managed platform with good PostgreSQL support
- **VPS**: Full control, cost-effective for larger deployments
- **Kubernetes**: For enterprise-scale deployments with high availability

---

## Deployment Options

### Deployment Decision Tree

```mermaid
graph TD
    Start[Start Deployment] --> Budget{Budget?}
    Budget -->|< $50/mo| VPS[VPS Deployment]
    Budget -->|$50-200/mo| Managed{Managed Platform?}
    Budget -->|> $200/mo| K8s[Kubernetes]
    
    Managed -->|Yes| Railway{Need Simple?}
    Managed -->|No| Render[Render Deployment]
    
    Railway -->|Yes| RailwayDeploy[Railway Deployment]
    Railway -->|No| Render
    
    VPS --> VPSDeploy[VPS Deployment Guide]
    K8s --> K8sDeploy[Kubernetes Deployment]
    RailwayDeploy --> Done[Deployment Complete]
    Render --> Done
    VPSDeploy --> Done
    K8sDeploy --> Done
    
    style Start fill:#e8f5e9
    style Done fill:#e8f5e9
    style VPS fill:#fff3e0
    style Railway fill:#e3f2fd
    style Render fill:#f3e5f5
    style K8s fill:#fce4ec
```

### Comparison

| Platform | Difficulty | Cost | Control | Scalability | Best For |
|----------|-----------|------|---------|-------------|----------|
| Railway | Easy | $$ | Low | Medium | Quick MVP, small teams |
| Render | Easy | $$ | Low | Medium | Simple apps, good Postgres |
| VPS | Medium | $ | High | High | Cost-effective, custom needs |
| Kubernetes | Hard | $$$ | High | Very High | Enterprise, high availability |

---

## Pre-Deployment Checklist

Before deploying to production, complete these steps:

### 1. Security Configuration

- [ ] Generate a secure `SECRET_KEY` using `openssl rand -hex 32`
- [ ] Set strong database passwords (minimum 16 characters, mixed case, numbers, symbols)
- [ ] Configure AI API keys with appropriate rate limits
- [ ] Enable HTTPS/TLS with valid SSL certificates
- [ ] Set up proper CORS origins (restrict to your domain only)
- [ ] Configure secure session cookie settings

### 2. Database Setup

- [ ] Choose a PostgreSQL hosting provider (Supabase, RDS, Neon, Railway)
- [ ] Create a production database
- [ ] Configure connection pooling settings
- [ ] Enable automated backups (daily minimum)
- [ ] Set up read replicas if expecting high traffic

### 3. Environment Variables

- [ ] Create production `.env` file with all required variables
- [ ] Never commit `.env` files to version control
- [ ] Use secret management service for sensitive values
- [ ] Test all environment variables in staging first

### 4. Domain & DNS

- [ ] Purchase and configure domain name
- [ ] Set up DNS records (A, CNAME, MX if needed)
- [ ] Configure SSL certificates (Let's Encrypt or custom)
- [ ] Set up CDN if using static asset hosting

### 5. Monitoring & Logging

- [ ] Set up error tracking (Sentry, Rollbar)
- [ ] Configure application logging
- [ ] Set up uptime monitoring (Pingdom, UptimeRobot)
- [ ] Configure alerting for critical failures

### 6. CI/CD Pipeline

- [ ] Set up automated testing on push
- [ ] Configure automated deployment on merge to main
- [ ] Set up rollback mechanism
- [ ] Configure deployment notifications

---

## Option 1: Railway Deployment

Railway is the easiest deployment option with built-in PostgreSQL and container support.

### Prerequisites

- Railway account (free tier available)
- GitHub account with repository connected to Railway

### Step 1: Connect Repository

1. Log in to [Railway](https://railway.app)
2. Click "New Project" → "Deploy from GitHub repo"
3. Select your TrustOS repository
4. Railway will detect the Docker Compose configuration

### Step 2: Configure PostgreSQL

1. Railway will automatically create a PostgreSQL service
2. Click on the PostgreSQL service
3. Copy the connection string
4. Add to your backend environment variables as `DATABASE_URL`

### Step 3: Configure Backend Service

1. Click on the backend service
2. Add environment variables:
   ```bash
   DATABASE_URL=postgresql+asyncpg://...
   SYNC_DATABASE_URL=postgresql://...
   SECRET_KEY=<generate with openssl rand -hex 32>
   OPENAI_API_KEY=sk-...
   AI_PROVIDER=openai
   STORAGE_PATH=/app/storage
   ```
3. Set the root directory to `backend`
4. Set the start command to `uvicorn app.main:app --host 0.0.0.0 --port $PORT`

### Step 4: Configure Frontend Service

1. Click "New Service" → "GitHub Repo"
2. Select the same repository
3. Set the root directory to `frontend`
4. Add environment variable:
   ```bash
   NEXT_PUBLIC_API_URL=https://your-backend-url.railway.app
   ```
5. Set the start command to `npm start`
6. Set the build command to `npm run build`

### Step 5: Deploy

1. Click "Deploy" on each service
2. Railway will build and deploy your services
3. Wait for the deployment to complete (2-5 minutes)
4. Access your application at the provided Railway URLs

### Step 6: Configure Custom Domain (Optional)

1. Click on your frontend service
2. Go to "Settings" → "Networking"
3. Add your custom domain
4. Update DNS records as instructed by Railway
5. Railway will automatically provision SSL certificates

### Railway-Specific Considerations

- **Storage**: Railway provides ephemeral storage, use Railway Volume for persistent storage
- **Database**: Railway PostgreSQL includes automated backups
- **Scaling**: Automatic scaling based on usage
- **Cost**: Free tier available, then $5/month per service

---

## Option 2: Render Deployment

Render offers excellent PostgreSQL support and simple deployment.

### Prerequisites

- Render account
- GitHub account with repository

### Step 1: Deploy PostgreSQL

1. Log in to [Render](https://render.com)
2. Click "New" → "PostgreSQL"
3. Choose a database name (e.g., `trustos-prod`)
4. Select a region closest to your users
5. Choose a plan (Free tier available)
6. Click "Create Database"
7. Copy the internal database URL

### Step 2: Deploy Backend

1. Click "New" → "Web Service"
2. Connect your GitHub repository
3. Set the following:
   - **Name**: `trustos-backend`
   - **Root Directory**: `backend`
   - **Build Command**: `pip install -r requirements.txt`
   - **Start Command**: `uvicorn app.main:app --host 0.0.0.0 --port $PORT`
4. Add environment variables:
   ```bash
   DATABASE_URL=<your-postgres-url>
   SYNC_DATABASE_URL=<your-postgres-url-sync>
   SECRET_KEY=<generate with openssl rand -hex 32>
   OPENAI_API_KEY=sk-...
   AI_PROVIDER=openai
   STORAGE_PATH=/opt/render/project/storage
   ```
5. Click "Create Web Service"

### Step 3: Deploy Frontend

1. Click "New" → "Web Service"
2. Connect your GitHub repository
3. Set the following:
   - **Name**: `trustos-frontend`
   - **Root Directory**: `frontend`
   - **Build Command**: `npm run build`
   - **Start Command**: `npm start`
4. Add environment variable:
   ```bash
   NEXT_PUBLIC_API_URL=https://trustos-backend.onrender.com
   ```
5. Click "Create Web Service"

### Step 4: Initialize Database

1. SSH into your backend service (Render provides this)
2. Run the seed script:
   ```bash
   python seed.py
   ```

### Step 5: Configure Custom Domain

1. Go to your frontend service settings
2. Add your custom domain
3. Update DNS records as instructed
4. Render will automatically provision SSL

### Render-Specific Considerations

- **Free Tier**: Available but spins down after inactivity
- **Database**: Automated daily backups included
- **Storage**: Use Render Disk for persistent storage
- **Cost**: Free tier available, then ~$7/month per service

---

## Option 3: VPS Deployment

For full control and cost-effectiveness, deploy to a VPS (DigitalOcean, Linode, AWS EC2, etc.).

### Prerequisites

- VPS with Ubuntu 22.04+ (minimum 2GB RAM, 2 CPU)
- Domain name
- SSH access to VPS
- Basic Linux command-line knowledge

### Step 1: Prepare VPS

```bash
# SSH into your VPS
ssh root@your-vps-ip

# Update system
apt update && apt upgrade -y

# Install Docker
curl -fsSL https://get.docker.com -o get-docker.sh
sh get-docker.sh

# Install Docker Compose
curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

# Install Nginx
apt install nginx -y

# Install Certbot for SSL
apt install certbot python3-certbot-nginx -y
```

### Step 2: Clone Repository

```bash
# Install Git
apt install git -y

# Clone repository
cd /opt
git clone https://gitea.thetempleofdoom.com/drjones/trustos.git
cd trustos
```

### Step 3: Configure Environment

```bash
# Copy environment template
cp backend/.env.example backend/.env

# Edit with production values
nano backend/.env
```

Set the following:
```bash
DATABASE_URL=postgresql+asyncpg://trustos:strong-password@postgres:5432/trustos
SYNC_DATABASE_URL=postgresql://trustos:strong-password@postgres:5432/trustos
SECRET_KEY=<generate with openssl rand -hex 32>
OPENAI_API_KEY=sk-...
AI_PROVIDER=openai
STORAGE_PATH=/app/storage
```

### Step 4: Configure Docker Compose

Create production docker-compose.yml:

```yaml
version: '3.8'

services:
  postgres:
    image: postgres:16
    container_name: trustos-postgres
    environment:
      POSTGRES_USER: trustos
      POSTGRES_PASSWORD: strong-password
      POSTGRES_DB: trustos
    volumes:
      - postgres_data:/var/lib/postgresql/data
    restart: unless-stopped
    networks:
      - trustos-network

  backend:
    build:
      context: .
      dockerfile: infra/Dockerfile.backend
    container_name: trustos-backend
    environment:
      DATABASE_URL: postgresql+asyncpg://trustos:strong-password@postgres:5432/trustos
      SYNC_DATABASE_URL: postgresql://trustos:strong-password@postgres:5432/trustos
      SECRET_KEY: ${SECRET_KEY}
      OPENAI_API_KEY: ${OPENAI_API_KEY}
      AI_PROVIDER: openai
      STORAGE_PATH: /app/storage
    depends_on:
      - postgres
    restart: unless-stopped
    networks:
      - trustos-network

  frontend:
    build:
      context: .
      dockerfile: infra/Dockerfile.frontend
    container_name: trustos-frontend
    environment:
      NEXT_PUBLIC_API_URL: https://api.yourdomain.com
    depends_on:
      - backend
    restart: unless-stopped
    networks:
      - trustos-network

volumes:
  postgres_data:

networks:
  trustos-network:
    driver: bridge
```

### Step 5: Start Services

```bash
# Start all services
docker-compose up -d

# Check status
docker-compose ps

# View logs
docker-compose logs -f
```

### Step 6: Initialize Database

```bash
# Run seed script
docker-compose exec backend python seed.py
```

### Step 7: Configure Nginx

Create Nginx configuration:

```bash
nano /etc/nginx/sites-available/trustos
```

Add the following:

```nginx
# Frontend (Next.js)
server {
    listen 80;
    server_name yourdomain.com www.yourdomain.com;

    location / {
        proxy_pass http://localhost:3000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}

# Backend API
server {
    listen 80;
    server_name api.yourdomain.com;

    location / {
        proxy_pass http://localhost:8000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}
```

Enable the site:

```bash
ln -s /etc/nginx/sites-available/trustos /etc/nginx/sites-enabled/
nginx -t
systemctl restart nginx
```

### Step 8: Configure SSL with Let's Encrypt

```bash
# Obtain SSL certificate
certbot --nginx -d yourdomain.com -d www.yourdomain.com -d api.yourdomain.com

# Certbot will automatically configure Nginx with SSL

# Test auto-renewal
certbot renew --dry-run
```

### Step 9: Set Up Firewall

```bash
# Configure UFW
ufw allow 22/tcp
ufw allow 80/tcp
ufw allow 443/tcp
ufw enable
```

### Step 10: Set Up Monitoring

Install monitoring tools:

```bash
# Install htop for resource monitoring
apt install htop -y

# Install fail2ban for security
apt install fail2ban -y

# Configure fail2ban
nano /etc/fail2ban/jail.local
```

### VPS-Specific Considerations

- **Backups**: Set up automated PostgreSQL backups to S3 or similar
- **Updates**: Regularly update system and Docker images
- **Security**: Use SSH keys, disable password authentication
- **Cost**: ~$20-50/month depending on VPS size

---

## Option 4: Kubernetes Deployment

For enterprise-scale deployments with high availability requirements.

### Prerequisites

- Kubernetes cluster (AWS EKS, GKE, AKS, or self-hosted)
- kubectl configured
- Helm installed
- Ingress controller (NGINX, Traefik, etc.)

### Step 1: Create Namespace

```bash
kubectl create namespace trustos
```

### Step 2: Create Secrets

```bash
# Create database secret
kubectl create secret generic trustos-db-secret \
  --from-literal=database-url="postgresql+asyncpg://..." \
  --namespace=trustos

# Create API secret
kubectl create secret generic trustos-api-secret \
  --from-literal=secret-key="..." \
  --from-literal=openai-api-key="sk-..." \
  --namespace=trustos
```

### Step 3: Deploy PostgreSQL

Using Helm:

```bash
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install trustos-postgres bitnami/postgresql \
  --namespace trustos \
  --set auth.password=strong-password \
  --set auth.database=trustos \
  --set persistence.enabled=true
```

### Step 4: Create Backend Deployment

```yaml
# backend-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: trustos-backend
  namespace: trustos
spec:
  replicas: 3
  selector:
    matchLabels:
      app: trustos-backend
  template:
    metadata:
      labels:
        app: trustos-backend
    spec:
      containers:
      - name: backend
        image: your-registry/trustos-backend:latest
        ports:
        - containerPort: 8000
        env:
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: trustos-db-secret
              key: database-url
        - name: SECRET_KEY
          valueFrom:
            secretKeyRef:
              name: trustos-api-secret
              key: secret-key
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "1Gi"
            cpu: "1000m"
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 5
          periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: trustos-backend
  namespace: trustos
spec:
  selector:
    app: trustos-backend
  ports:
  - port: 8000
    targetPort: 8000
  type: ClusterIP
```

### Step 5: Create Frontend Deployment

```yaml
# frontend-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: trustos-frontend
  namespace: trustos
spec:
  replicas: 2
  selector:
    matchLabels:
      app: trustos-frontend
  template:
    metadata:
      labels:
        app: trustos-frontend
    spec:
      containers:
      - name: frontend
        image: your-registry/trustos-frontend:latest
        ports:
        - containerPort: 3000
        env:
        - name: NEXT_PUBLIC_API_URL
          value: "https://api.yourdomain.com"
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"
---
apiVersion: v1
kind: Service
metadata:
  name: trustos-frontend
  namespace: trustos
spec:
  selector:
    app: trustos-frontend
  ports:
  - port: 3000
    targetPort: 3000
  type: ClusterIP
```

### Step 6: Create Ingress

```yaml
# ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: trustos-ingress
  namespace: trustos
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  tls:
  - hosts:
    - yourdomain.com
    - api.yourdomain.com
    secretName: trustos-tls
  rules:
  - host: yourdomain.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: trustos-frontend
            port:
              number: 3000
  - host: api.yourdomain.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: trustos-backend
            port:
              number: 8000
```

### Step 7: Deploy

```bash
kubectl apply -f backend-deployment.yaml
kubectl apply -f frontend-deployment.yaml
kubectl apply -f ingress.yaml
```

### Step 8: Initialize Database

```bash
kubectl exec -it deployment/trustos-backend -n trustos -- python seed.py
```

### Kubernetes-Specific Considerations

- **High Availability**: Multiple replicas across availability zones
- **Auto-scaling**: Configure Horizontal Pod Autoscaler
- **Secrets Management**: Use external secret manager (Vault, AWS Secrets Manager)
- **Monitoring**: Install Prometheus and Grafana
- **Logging**: Use centralized logging (ELK stack, Loki)
- **Cost**: Higher due to infrastructure complexity

---

## Post-Deployment Steps

### 1. Verify Deployment

```bash
# Check health endpoint
curl https://api.yourdomain.com/health

# Check frontend
curl https://yourdomain.com

# Check API documentation
open https://api.yourdomain.com/docs
```

### 2. Run Database Migrations

```bash
# If using Alembic
alembic upgrade head
```

### 3. Seed Initial Data

```bash
# Run seed script
python seed.py
```

### 4. Configure Monitoring

Set up monitoring for:

- Application health (uptime, response times)
- Database performance (query times, connection pool)
- Resource usage (CPU, memory, disk)
- Error rates (500 errors, exceptions)

### 5. Set Up Alerts

Configure alerts for:

- Application downtime
- High error rates
- Database connection issues
- Disk space usage > 80%
- Memory usage > 90%

### 6. Test User Flows

Test critical user journeys:

- User login
- Dashboard loading
- Finding creation
- Report generation
- AI translation

---

## Monitoring & Maintenance

### Application Monitoring

#### Recommended Tools

- **Sentry**: Error tracking and performance monitoring
- **Datadog**: Full-stack monitoring
- **New Relic**: Application performance monitoring
- **Prometheus + Grafana**: Open-source monitoring stack

#### Key Metrics to Monitor

- Request rate and response time
- Error rate (4xx, 5xx)
- Database query performance
- AI API usage and costs
- Active users and sessions

### Log Management

#### Recommended Tools

- **Loggly**: Cloud-based log management
- **Papertrail**: Simple log aggregation
- **ELK Stack**: Elasticsearch, Logstash, Kibana
- **Loki**: Grafana's log aggregation system

#### Log Levels

- **ERROR**: Critical errors requiring immediate attention
- **WARNING**: Issues that should be investigated
- **INFO**: Normal operational information
- **DEBUG**: Detailed debugging information (development only)

### Regular Maintenance Tasks

#### Daily

- Review error logs
- Check system resource usage
- Verify backup completion

#### Weekly

- Review security logs
- Check for dependency updates
- Review performance metrics

#### Monthly

- Apply security patches
- Review and optimize database queries
- Review AI API costs
- Test disaster recovery procedures

#### Quarterly

- Security audit
- Performance review
- Capacity planning
- Disaster recovery testing

---

## Backup & Recovery

### Database Backups

#### PostgreSQL Backup Strategy

```bash
# Daily backup script
#!/bin/bash
DATE=$(date +%Y%m%d)
BACKUP_DIR="/backups/postgres"
pg_dump -h localhost -U trustos trustos | gzip > $BACKUP_DIR/trustos_$DATE.sql.gz

# Keep last 30 days
find $BACKUP_DIR -name "trustos_*.sql.gz" -mtime +30 -delete
```

#### Backup Storage

- Store backups in multiple locations (local + cloud)
- Use S3, Glacier, or similar for cloud storage
- Encrypt backups at rest
- Test restore procedures regularly

### Application Backups

#### What to Back Up

- Database dumps
- File storage (PDF reports, uploads)
- Configuration files
- Environment variables (secure storage)

#### Backup Schedule

- **Database**: Daily, retain 30 days
- **Files**: Weekly, retain 90 days
- **Config**: On change, version control

### Recovery Procedures

#### Database Recovery

```bash
# Restore from backup
gunzip < trustos_20240101.sql.gz | psql -h localhost -U trustos trustos
```

#### Disaster Recovery

1. Identify the scope of the disaster
2. Determine the point of recovery
3. Restore from the most recent good backup
4. Verify data integrity
5. Test application functionality
6. Monitor for issues

---

## Troubleshooting

### Common Issues

#### Application Won't Start

**Symptoms**: Container exits immediately, 500 errors

**Solutions**:
- Check logs: `docker-compose logs backend`
- Verify environment variables
- Check database connectivity
- Verify port availability

#### Database Connection Errors

**Symptoms**: "could not connect to server"

**Solutions**:
- Verify database is running
- Check connection string
- Verify network connectivity
- Check firewall rules

#### SSL Certificate Issues

**Symptoms**: Browser warnings, certificate errors

**Solutions**:
- Verify DNS records are correct
- Check certificate expiration
- Renew with Certbot: `certbot renew`
- Verify Nginx configuration

#### High Memory Usage

**Symptoms**: OOM errors, slow performance

**Solutions**:
- Check for memory leaks
- Increase container memory limits
- Optimize database queries
- Enable connection pooling

#### AI Features Not Working

**Symptoms**: Empty AI translations, errors

**Solutions**:
- Verify API key is valid
- Check API quota/credits
- Review AI service logs
- Test API key manually

### Getting Help

1. Check logs: `docker-compose logs -f`
2. Review documentation: `/docs`
3. Check status page: `https://status.trustos.com`
4. Contact support: `support@trustos.com`

---

## Security Best Practices

### Network Security

- Use HTTPS everywhere
- Configure firewall rules
- Use VPN for admin access
- Implement rate limiting
- Use DDoS protection

### Application Security

- Keep dependencies updated
- Use strong secrets
- Enable security headers
- Implement CORS properly
- Regular security audits

### Data Security

- Encrypt data at rest
- Encrypt data in transit
- Implement access controls
- Regular security training
- Incident response plan

---

## Cost Optimization

### Railway

- Use free tier for development
- Scale down during off-hours
- Monitor usage regularly
- Use reserved instances for production

### Render

- Use free tier for staging
- Optimize build times
- Use appropriate instance sizes
- Monitor resource usage

### VPS

- Right-size your instance
- Use spot instances for non-critical workloads
- Implement auto-scaling
- Monitor and optimize resource usage

### Kubernetes

- Use resource limits
- Implement auto-scaling
- Use spot instances
- Optimize pod requests/limits

---

## Conclusion

This deployment guide covers the most common deployment scenarios for TrustOS. Choose the option that best fits your team's expertise, budget, and scalability requirements.

For additional support or questions, refer to the main [README.md](../README.md) or contact the TrustOS development team.