Skip to content

The Pipeline

The pipeline is AMP's execution engine. It transforms missions into published, optimized content through six sequential stages.

Overview

graph LR
    I[Intake] --> S[Strategy]
    S --> C[Content]
    C --> P[Publish]
    P --> A[Analytics]
    A --> O[Optimize]
    O -.->|Feedback Loop| S

Each stage has a specific responsibility and produces output consumed by subsequent stages.

Pipeline Characteristics

Property Description
Sequential Stages execute in order; each depends on the previous
Resumable Can restart from any stage after failure
Observable Real-time progress tracking via API
Retryable Automatic retry with exponential backoff
Rollbackable Previous stages can be undone on failure

Stage Details

1. Intake

Purpose: Parse and validate mission parameters

Input:

  • Mission definition (objectives, platforms, constraints, KPIs)
  • Brand context reference

Processing:

  1. Validate mission schema
  2. Resolve brand context
  3. Check platform availability
  4. Verify budget constraints
  5. Build execution context

Output:

{
  "stage": "intake",
  "status": "completed",
  "output": {
    "mission_id": "msn_xxx",
    "brand_context_id": "brc_xxx",
    "validated_platforms": ["twitter", "linkedin"],
    "execution_context": {
      "start_date": "2024-01-15",
      "end_date": "2024-02-14",
      "total_posts_planned": 60,
      "daily_cadence": 2
    }
  }
}

Duration: 1-2 seconds

2. Strategy

Purpose: Develop content strategy and editorial calendar

Input:

  • Execution context from Intake
  • Brand context (voice, topics, competitors)
  • Historical performance data (if available)

Processing:

  1. Analyze objectives for content themes
  2. Map topics to content pillars
  3. Determine content mix (post types, formats)
  4. Create editorial calendar
  5. Assign optimal posting times per platform

Output:

{
  "stage": "strategy",
  "status": "completed",
  "output": {
    "content_pillars": [
      {"name": "Thought Leadership", "weight": 0.4},
      {"name": "Product Updates", "weight": 0.3},
      {"name": "Community Engagement", "weight": 0.3}
    ],
    "content_calendar": [
      {
        "date": "2024-01-16",
        "slots": [
          {
            "time": "09:00",
            "platform": "twitter",
            "pillar": "Thought Leadership",
            "topic": "API Design Patterns",
            "format": "thread"
          },
          {
            "time": "14:00",
            "platform": "linkedin",
            "pillar": "Thought Leadership",
            "topic": "API Design Patterns",
            "format": "article"
          }
        ]
      }
    ],
    "tone_guidelines": "Professional, technically accurate, conversational"
  }
}

Duration: 30-60 seconds

LLM Usage: This stage makes 1-3 LLM calls depending on mission complexity.

3. Content

Purpose: Generate copy and visuals for each calendar slot

Input:

  • Content calendar from Strategy
  • Brand context (voice, visual identity, examples)
  • Any linked source materials

Processing:

  1. For each calendar slot:
  2. Generate copy (headline, body, CTA)
  3. Create platform-specific variations
  4. Generate or select visuals
  5. Add hashtags and mentions
  6. Create A/B variations (optional)
  7. Validate content against constraints
  8. Check for brand guideline compliance

Output:

{
  "stage": "content",
  "status": "completed",
  "output": {
    "content_items": [
      {
        "id": "cnt_xxx",
        "calendar_slot": "2024-01-16T09:00:00Z",
        "platform": "twitter",
        "type": "thread",
        "copy": {
          "parts": [
            "APIs shouldn't feel like puzzles. Here's what we learned rebuilding ours from scratch. 🧵",
            "1/ The old way: 47 endpoints, inconsistent naming, three different auth methods. Developers were confused.",
            "2/ The new way: 12 endpoints, RESTful conventions, single OAuth2 flow. Support tickets dropped 73%."
          ],
          "hashtags": ["#API", "#DevEx"]
        },
        "visuals": [],
        "metadata": {
          "pillar": "Thought Leadership",
          "topic": "API Design Patterns",
          "confidence": 0.92
        }
      }
    ],
    "total_items": 60,
    "total_cost_cents": 2340
  }
}

Duration: 2-5 minutes (depends on content volume)

LLM Usage: 1 call per content item, plus additional calls for visuals.

4. Publish

Purpose: Schedule and distribute content to platforms

Input:

  • Generated content from Content stage
  • Platform credentials (via Metricool)
  • Scheduling preferences

Processing:

  1. Upload media assets to CDN
  2. Format content for each platform's requirements
  3. Submit to Metricool for scheduling
  4. Record publishing metadata
  5. Handle any platform-specific validation

Output:

{
  "stage": "publish",
  "status": "completed",
  "output": {
    "scheduled": [
      {
        "content_id": "cnt_xxx",
        "platform": "twitter",
        "scheduled_for": "2024-01-16T09:00:00Z",
        "metricool_post_id": "mc_12345",
        "preview_url": "https://preview.metricool.com/..."
      }
    ],
    "total_scheduled": 60,
    "first_publish_at": "2024-01-16T09:00:00Z",
    "last_publish_at": "2024-02-14T14:00:00Z"
  }
}

Duration: 10-30 seconds

5. Analytics

Purpose: Collect performance data from published content

Input:

  • Published content references
  • Platform analytics APIs (via Metricool)

Processing:

  1. Poll for engagement metrics
  2. Aggregate by content item, platform, time
  3. Compare against KPI targets
  4. Identify trends and patterns
  5. Flag underperforming content

Output:

{
  "stage": "analytics",
  "status": "running",
  "output": {
    "period": "2024-01-16/2024-01-23",
    "metrics": {
      "total_impressions": 45678,
      "total_engagements": 1823,
      "engagement_rate": 3.99,
      "link_clicks": 234,
      "new_followers": 56
    },
    "kpi_status": [
      {
        "metric": "engagement_rate",
        "target": 4.0,
        "current": 3.99,
        "status": "on_track"
      }
    ],
    "top_performers": ["cnt_xxx", "cnt_yyy"],
    "underperformers": ["cnt_zzz"]
  }
}

Duration: Continuous (runs periodically while mission is active)

6. Optimize

Purpose: Analyze results and improve future content

Input:

  • Analytics data from Analytics stage
  • Historical performance patterns
  • Original mission objectives

Processing:

  1. Identify high-performing content characteristics
  2. Detect timing patterns (best days, hours)
  3. Analyze topic and format effectiveness
  4. Generate recommendations
  5. Update strategy parameters for next cycle

Output:

{
  "stage": "optimize",
  "status": "completed",
  "output": {
    "insights": [
      {
        "type": "timing",
        "finding": "Tuesday 2pm posts outperform Monday 9am by 2.3x",
        "confidence": 0.87,
        "action": "shift_schedule"
      },
      {
        "type": "content",
        "finding": "Threads with code snippets get 40% more engagement",
        "confidence": 0.91,
        "action": "increase_code_content"
      }
    ],
    "strategy_updates": {
      "timing_adjustments": {"tuesday_weight": 1.5},
      "content_adjustments": {"code_snippet_frequency": 0.6}
    },
    "predicted_improvement": {
      "engagement_rate": "+0.5%"
    }
  }
}

Duration: 1-2 minutes

Feedback Loop: Strategy updates feed back into the Strategy stage for the next content generation cycle.

Job Lifecycle

stateDiagram-v2
    [*] --> pending: Job Created
    pending --> running: Worker Claims
    running --> completed: All Stages Done
    running --> failed: Unrecoverable Error
    running --> retrying: Transient Error
    retrying --> running: Retry Attempt
    retrying --> failed: Max Retries Exceeded
    failed --> [*]
    completed --> [*]

Job States

State Description
pending Job created, waiting for worker
running Worker processing stages
retrying Failed, will retry
completed All stages finished successfully
failed Unrecoverable error, no more retries
cancelled Manually stopped

Error Handling

Transient Errors

Automatically retried with exponential backoff:

  • Network timeouts
  • Rate limiting (429)
  • Provider temporary unavailability
  • Database connection errors

Retry configuration:

MaxRetries:    3
InitialDelay:  2s
MaxDelay:      30s
Multiplier:    2.0

Permanent Errors

Cause immediate failure:

  • Invalid mission configuration
  • Authentication failures
  • Missing required data
  • Validation errors

Stage Rollback

When a stage fails and rollback is enabled, previous stages undo their changes:

sequenceDiagram
    participant Intake
    participant Strategy
    participant Content

    Intake->>Strategy: Stage Complete
    Strategy->>Content: Stage Complete
    Content->>Content: Failure!
    Content->>Strategy: Rollback
    Strategy->>Intake: Rollback
    Note over Intake,Content: Job marked as failed

Monitoring Jobs

Get Job Status

curl https://api.amp.dev/v1/pipeline/jobs/job_xxx \
  -H "Authorization: Bearer $AMP_API_KEY"

Stream Job Logs

curl https://api.amp.dev/v1/pipeline/jobs/job_xxx/logs?stream=true \
  -H "Authorization: Bearer $AMP_API_KEY"

Retry Failed Job

curl -X POST https://api.amp.dev/v1/pipeline/jobs/job_xxx/retry \
  -H "Authorization: Bearer $AMP_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"from_stage": "content"}'

Performance Optimization

Parallelization

Content generation parallelizes within the Content stage:

Calendar Slots: [slot1, slot2, slot3, slot4, ...]
                   │       │       │       │
                   ▼       ▼       ▼       ▼
Workers:        [gen1]  [gen2]  [gen3]  [gen4]
                   │       │       │       │
                   └───────┴───────┴───────┘
                              │
                              ▼
                      Aggregate Results

Caching

  • Brand context cached for pipeline duration
  • LLM responses cached for identical prompts
  • Media assets cached in CDN

Batch Operations

Database operations are batched:

  • Content items written in batches of 50
  • Analytics metrics aggregated before insert
  • Publishing confirmations bulk-updated