River Phase 5: Deploy Cloud Run Job + Scheduler

{
  "code": "## River Phase 5: Cloud Run Job + Scheduler Deployment\n\n### Architecture\n\n```\nCloud Scheduler → Cloud Run Job → river-worker.py → Supabase\n    (cron)           (serverless)     (pipeline)      (data)\n```\n\n### Cloud Run Job Configuration\n\n**Job Name**: `river-worker`\n**Region**: `us-central1` (close to Supabase Ohio)\n**Image**: `gcr.io/duty-pay/river-worker:latest`\n**Memory**: 512Mi\n**CPU**: 1\n**Timeout**: 3600s (1 hour max)\n**Max Retries**: 2\n**Parallelism**: 1 (serial phases)\n\n### Dockerfile\n```dockerfile\nFROM python:3.12-slim\nWORKDIR /app\nCOPY requirements-river.txt .\nRUN pip install --no-cache-dir -r requirements-river.txt\nCOPY c8_platform/river_worker.py .\nCOPY c8_platform/rosetta_core/ ./rosetta_core/\nCOPY c8_platform/rosetta_mcp.py .\nENV PYTHONUNBUFFERED=1\nENTRYPOINT [\"python\", \"river_worker.py\"]\n```\n\n### Cloud Scheduler Cron Jobs\n\n| Job Name | Schedule | Phase | Command |\n|----------|----------|-------|---------|\n| `river-ingest` | `0 */6 * * *` | Ingest | `--phase ingest` |\n| `river-classify` | `0 */4 * * *` | Classify | `--phase classify` |\n| `river-enrich` | `0 2 * * *` | Enrich | `--phase enrich` |\n| `river-learn` | `0 4 * * *` | Learn | `--phase learn` |\n\n### Deploy Commands\n```bash\n# 1. Build and push image\ngcloud builds submit --tag gcr.io/duty-pay/river-worker:latest \\\n  --project duty-pay \\\n  -f Dockerfile.river\n\n# 2. Create Cloud Run Job\ngcloud run jobs create river-worker \\\n  --image gcr.io/duty-pay/river-worker:latest \\\n  --region us-central1 \\\n  --memory 512Mi \\\n  --cpu 1 \\\n  --max-retries 2 \\\n  --task-timeout 3600s \\\n  --set-env-vars \"SUPABASE_URL=$SUPABASE_URL,SUPABASE_KEY=$SUPABASE_KEY,ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY\" \\\n  --project duty-pay\n\n# 3. Create Cloud Scheduler jobs\n# Ingest every 6 hours\ngcloud scheduler jobs create http river-ingest \\\n  --location us-central1 \\\n  --schedule \"0 */6 * * *\" \\\n  --uri \"https://us-central1-run.googleapis.com/apis/run.googleapis.com/v1/namespaces/duty-pay/jobs/river-worker:run\" \\\n  --http-method POST \\\n  --headers \"Content-Type=application/json\" \\\n  --message-body '{\"overrides\":{\"containerOverrides\":[{\"args\":[\"--phase\",\"ingest\"]}]}}' \\\n  --oauth-service-account-email river-worker@duty-pay.iam.gserviceaccount.com \\\n  --project duty-pay\n\n# Classify every 4 hours\ngcloud scheduler jobs create http river-classify \\\n  --location us-central1 \\\n  --schedule \"0 */4 * * *\" \\\n  --uri \"https://us-central1-run.googleapis.com/apis/run.googleapis.com/v1/namespaces/duty-pay/jobs/river-worker:run\" \\\n  --http-method POST \\\n  --headers \"Content-Type=application/json\" \\\n  --message-body '{\"overrides\":{\"containerOverrides\":[{\"args\":[\"--phase\",\"classify\"]}]}}' \\\n  --oauth-service-account-email river-worker@duty-pay.iam.gserviceaccount.com \\\n  --project duty-pay\n\n# Enrich daily at 2 AM UTC\ngcloud scheduler jobs create http river-enrich \\\n  --location us-central1 \\\n  --schedule \"0 2 * * *\" \\\n  --uri \"https://us-central1-run.googleapis.com/apis/run.googleapis.com/v1/namespaces/duty-pay/jobs/river-worker:run\" \\\n  --http-method POST \\\n  --headers \"Content-Type=application/json\" \\\n  --message-body '{\"overrides\":{\"containerOverrides\":[{\"args\":[\"--phase\",\"enrich\"]}]}}' \\\n  --oauth-service-account-email river-worker@duty-pay.iam.gserviceaccount.com \\\n  --project duty-pay\n\n# Learn daily at 4 AM UTC\ngcloud scheduler jobs create http river-learn \\\n  --location us-central1 \\\n  --schedule \"0 4 * * *\" \\\n  --uri \"https://us-central1-run.googleapis.com/apis/run.googleapis.com/v1/namespaces/duty-pay/jobs/river-worker:run\" \\\n  --http-method POST \\\n  --headers \"Content-Type=application/json\" \\\n  --message-body '{\"overrides\":{\"containerOverrides\":[{\"args\":[\"--phase\",\"learn\"]}]}}' \\\n  --oauth-service-account-email river-worker@duty-pay.iam.gserviceaccount.com \\\n  --project duty-pay\n```\n\n### IAM Setup\n```bash\n# Create service account\ngcloud iam service-accounts create river-worker \\\n  --display-name \"River Worker\" --project duty-pay\n\n# Grant Cloud Run Job invoker\ngcloud run jobs add-iam-policy-binding river-worker \\\n  --member serviceAccount:river-worker@duty-pay.iam.gserviceaccount.com \\\n  --role roles/run.invoker \\\n  --region us-central1 --project duty-pay\n\n# Grant Secret Manager access (for API keys)\ngcloud projects add-iam-policy-binding duty-pay \\\n  --member serviceAccount:river-worker@duty-pay.iam.gserviceaccount.com \\\n  --role roles/secretmanager.secretAccessor\n```\n\n### Monitoring\n```bash\n# View recent executions\ngcloud run jobs executions list --job river-worker --region us-central1\n\n# View logs\ngcloud logging read \"resource.type=cloud_run_job AND resource.labels.job_name=river-worker\" \\\n  --limit 50 --project duty-pay\n\n# Alert on failure (Cloud Monitoring)\ngcloud alpha monitoring policies create \\\n  --display-name \"River Worker Failure Alert\" \\\n  --condition-display-name \"Job Failed\" \\\n  --condition-filter 'resource.type=\"cloud_run_job\" AND metric.type=\"run.googleapis.com/job/completed_task_attempt_count\" AND metric.labels.result=\"failed\"' \\\n  --notification-channels $SLACK_CHANNEL_ID\n```\n\n### Cost Estimate\n- Cloud Run Job: ~$0.50/day (4-6 executions × 5-15 min each)\n- Cloud Scheduler: Free (< 3 jobs/month free tier, $0.10/job/month after)\n- Total: ~$15-20/month\n\n### Prerequisites for Desktop Lu\n1. `river_worker.py` exists with `--phase` CLI arg parsing\n2. Service account created with Supabase + Secret Manager access\n3. Supabase URL/Key in Secret Manager\n4. Docker image builds successfully\n5. Test locally: `python river_worker.py --phase ingest` before deploying",
  "content": "## River Phase 5: Cloud Run Job + Scheduler Deployment\n\n### Architecture\n\n```\nCloud Scheduler → Cloud Run Job → river-worker.py → Supabase\n    (cron)           (serverless)     (pipeline)      (data)\n```\n\n### Cloud Run Job Configuration\n\n**Job Name**: `river-worker`\n**Region**: `us-central1` (close to Supabase Ohio)\n**Image**: `gcr.io/duty-pay/river-worker:latest`\n**Memory**: 512Mi\n**CPU**: 1\n**Timeout**: 3600s (1 hour max)\n**Max Retries**: 2\n**Parallelism**: 1 (serial phases)\n\n### Dockerfile\n```dockerfile\nFROM python:3.12-slim\nWORKDIR /app\nCOPY requirements-river.txt .\nRUN pip install --no-cache-dir -r requirements-river.txt\nCOPY c8_platform/river_worker.py .\nCOPY c8_platform/rosetta_core/ ./rosetta_core/\nCOPY c8_platform/rosetta_mcp.py .\nENV PYTHONUNBUFFERED=1\nENTRYPOINT [\"python\", \"river_worker.py\"]\n```\n\n### Cloud Scheduler Cron Jobs\n\n| Job Name | Schedule | Phase | Command |\n|----------|----------|-------|---------|\n| `river-ingest` | `0 */6 * * *` | Ingest | `--phase ingest` |\n| `river-classify` | `0 */4 * * *` | Classify | `--phase classify` |\n| `river-enrich` | `0 2 * * *` | Enrich | `--phase enrich` |\n| `river-learn` | `0 4 * * *` | Learn | `--phase learn` |\n\n### Deploy Commands\n```bash\n# 1. Build and push image\ngcloud builds submit --tag gcr.io/duty-pay/river-worker:latest \\\n  --project duty-pay \\\n  -f Dockerfile.river\n\n# 2. Create Cloud Run Job\ngcloud run jobs create river-worker \\\n  --image gcr.io/duty-pay/river-worker:latest \\\n  --region us-central1 \\\n  --memory 512Mi \\\n  --cpu 1 \\\n  --max-retries 2 \\\n  --task-timeout 3600s \\\n  --set-env-vars \"SUPABASE_URL=$SUPABASE_URL,SUPABASE_KEY=$SUPABASE_KEY,ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY\" \\\n  --project duty-pay\n\n# 3. Create Cloud Scheduler jobs\n# Ingest every 6 hours\ngcloud scheduler jobs create http river-ingest \\\n  --location us-central1 \\\n  --schedule \"0 */6 * * *\" \\\n  --uri \"https://us-central1-run.googleapis.com/apis/run.googleapis.com/v1/namespaces/duty-pay/jobs/river-worker:run\" \\\n  --http-method POST \\\n  --headers \"Content-Type=application/json\" \\\n  --message-body '{\"overrides\":{\"containerOverrides\":[{\"args\":[\"--phase\",\"ingest\"]}]}}' \\\n  --oauth-service-account-email river-worker@duty-pay.iam.gserviceaccount.com \\\n  --project duty-pay\n\n# Classify every 4 hours\ngcloud scheduler jobs create http river-classify \\\n  --location us-central1 \\\n  --schedule \"0 */4 * * *\" \\\n  --uri \"https://us-central1-run.googleapis.com/apis/run.googleapis.com/v1/namespaces/duty-pay/jobs/river-worker:run\" \\\n  --http-method POST \\\n  --headers \"Content-Type=application/json\" \\\n  --message-body '{\"overrides\":{\"containerOverrides\":[{\"args\":[\"--phase\",\"classify\"]}]}}' \\\n  --oauth-service-account-email river-worker@duty-pay.iam.gserviceaccount.com \\\n  --project duty-pay\n\n# Enrich daily at 2 AM UTC\ngcloud scheduler jobs create http river-enrich \\\n  --location us-central1 \\\n  --schedule \"0 2 * * *\" \\\n  --uri \"https://us-central1-run.googleapis.com/apis/run.googleapis.com/v1/namespaces/duty-pay/jobs/river-worker:run\" \\\n  --http-method POST \\\n  --headers \"Content-Type=application/json\" \\\n  --message-body '{\"overrides\":{\"containerOverrides\":[{\"args\":[\"--phase\",\"enrich\"]}]}}' \\\n  --oauth-service-account-email river-worker@duty-pay.iam.gserviceaccount.com \\\n  --project duty-pay\n\n# Learn daily at 4 AM UTC\ngcloud scheduler jobs create http river-learn \\\n  --location us-central1 \\\n  --schedule \"0 4 * * *\" \\\n  --uri \"https://us-central1-run.googleapis.com/apis/run.googleapis.com/v1/namespaces/duty-pay/jobs/river-worker:run\" \\\n  --http-method POST \\\n  --headers \"Content-Type=application/json\" \\\n  --message-body '{\"overrides\":{\"containerOverrides\":[{\"args\":[\"--phase\",\"learn\"]}]}}' \\\n  --oauth-service-account-email river-worker@duty-pay.iam.gserviceaccount.com \\\n  --project duty-pay\n```\n\n### IAM Setup\n```bash\n# Create service account\ngcloud iam service-accounts create river-worker \\\n  --display-name \"River Worker\" --project duty-pay\n\n# Grant Cloud Run Job invoker\ngcloud run jobs add-iam-policy-binding river-worker \\\n  --member serviceAccount:river-worker@duty-pay.iam.gserviceaccount.com \\\n  --role roles/run.invoker \\\n  --region us-central1 --project duty-pay\n\n# Grant Secret Manager access (for API keys)\ngcloud projects add-iam-policy-binding duty-pay \\\n  --member serviceAccount:river-worker@duty-pay.iam.gserviceaccount.com \\\n  --role roles/secretmanager.secretAccessor\n```\n\n### Monitoring\n```bash\n# View recent executions\ngcloud run jobs executions list --job river-worker --region us-central1\n\n# View logs\ngcloud logging read \"resource.type=cloud_run_job AND resource.labels.job_name=river-worker\" \\\n  --limit 50 --project duty-pay\n\n# Alert on failure (Cloud Monitoring)\ngcloud alpha monitoring policies create \\\n  --display-name \"River Worker Failure Alert\" \\\n  --condition-display-name \"Job Failed\" \\\n  --condition-filter 'resource.type=\"cloud_run_job\" AND metric.type=\"run.googleapis.com/job/completed_task_attempt_count\" AND metric.labels.result=\"failed\"' \\\n  --notification-channels $SLACK_CHANNEL_ID\n```\n\n### Cost Estimate\n- Cloud Run Job: ~$0.50/day (4-6 executions × 5-15 min each)\n- Cloud Scheduler: Free (< 3 jobs/month free tier, $0.10/job/month after)\n- Total: ~$15-20/month\n\n### Prerequisites for Desktop Lu\n1. `river_worker.py` exists with `--phase` CLI arg parsing\n2. Service account created with Supabase + Secret Manager access\n3. Supabase URL/Key in Secret Manager\n4. Docker image builds successfully\n5. Test locally: `python river_worker.py --phase ingest` before deploying"
}