GitHub Integration - Langfuse

GitHub Integration for Langfuse Prompts

There are two methods to integrate Langfuse prompts with GitHub:

GitHub Repository Dispatch - Trigger CI/CD workflows when prompts change. This does not require additional infrastructure.
Sync Langfuse Prompts to a repository - Store prompts in a specific file in your repository. This involves a webhook server that listens for prompt version changes and commits them to the repository.

Trigger GitHub Actions

Trigger GitHub Actions workflows when Langfuse prompts change using repository_dispatch events.

sequenceDiagram
    participant User as User/Team
    participant LF as Langfuse
    participant GH as GitHub API
    participant Actions as GitHub Actions

User->>LF: Update prompt in Langfuse
    LF->>GH: POST /repos/owner/repo/dispatches
    GH->>Actions: Trigger repository_dispatch event
    Actions->>Actions: Run CI workflow (tests, deploy, etc.)
    Note over User,Actions: Prompt changes trigger automated workflows

1. Create GitHub Workflow

.github/workflows/langfuse-ci.yml:

name: Langfuse Prompt CI
on:
  repository_dispatch:
    types: [langfuse-prompt-update]
  workflow_dispatch:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run tests
        run: |
          echo "Testing prompt: ${{ github.event.client_payload.prompt.name }} v${{ github.event.client_payload.prompt.version }}"
          # Add your test commands
          # npm test
          # python -m pytest

deploy:
    needs: test
    runs-on: ubuntu-latest
    if: contains(github.event.client_payload.prompt.labels, 'production')
    steps:
      - uses: actions/checkout@v4
      - name: Deploy to production
        run: |
          echo "Deploying ${{ github.event.client_payload.prompt.name }} v${{ github.event.client_payload.prompt.version }}"
          # Your deployment commands

Accessing webhook data: Use github.event.client_payload.* to access prompt data:

# Example: Access webhook data in your workflow
- name: Process prompt data
  run: |
    echo "Action: ${{ github.event.client_payload.action }}"
    echo "Prompt: ${{ github.event.client_payload.prompt.name }}"
    echo "Version: ${{ github.event.client_payload.prompt.version }}"
    echo "Labels: ${{ github.event.client_payload.prompt.labels }}"

- name: Deploy only production prompts
  if: contains(github.event.client_payload.prompt.labels, 'production')
  run: echo "Deploying production prompt"

2. Create GitHub Token for Actions

Steps:

GitHub Settings > Developer settings > Personal access tokens
Generate new token (classic or fine-grained)
Select scope (see table below)

Token Type	Required Permissions
Personal Access Token (classic)	`repo` scope (public repos) or `public_repo` scope (private repos)
Fine-grained PAT or GitHub App	`read` and `write` to `actions`

3. Configure GitHub Action in Langfuse

Go to Prompts > Automations in your Langfuse project.
Click Create Automation.
Select GitHub Repository Dispatch.
Configure the automation:
- Dispatch URL: https://api.github.com/repos/{owner}/{repo}/dispatches (replace {owner} and {repo} with your values)
- Event Type: langfuse-prompt-update (must match the type in your GitHub workflow)
- GitHub Token: Enter your GitHub Personal Access Token. It will be stored securely.

4. Test GitHub Actions Integration

Update a prompt in Langfuse with the production label
Check GitHub Actions tab for triggered workflow
Verify that both test and deploy jobs run successfully

Sync Langfuse Prompts to a repository

Automatically sync prompt changes from Langfuse to GitHub using Prompt Version Webhooks. This enables version control for prompts and can trigger CI/CD workflows.

Overview of the Sync Workflow

Whenever you save a new prompt version in Langfuse, it's automatically committed to your GitHub repository. With this setup, you can also trigger CI/CD workflows when prompts change.

sequenceDiagram
    participant User as User/Team
    participant LF as Langfuse
    participant FastAPI as FastAPI Server
    participant GitHub as GitHub

User->>LF: Set up webhooks
    User->>LF: Modify a prompt
    LF->>FastAPI: POST /webhook/prompt (JSON payload)
    FastAPI->>GitHub: GET file SHA (if exists)
    GitHub-->>FastAPI: Return current file SHA
    FastAPI->>GitHub: PUT /repos/:owner/:repo/contents/:path
    GitHub->>GitHub: Create/update commit with prompt
    GitHub-->>FastAPI: ✅ Commit successful
    FastAPI-->>LF: 201 Created response
    Note over User,GitHub: Prompt changes now version-controlled in GitHub

Prerequisites for Sync

Langfuse Project: Prompt setup with Project Owner access
GitHub Repository: Public or private repo to store prompts
GitHub PAT: Personal Access Token with minimum required permissions (see Step 2 for details)
Python 3.9+ (for the example below, can be any language) with FastAPI, Uvicorn, httpx, Pydantic
Public HTTPS endpoint for your webhook server (Render, Fly.io, Heroku, etc.)

Step 1: Configure a Prompt Webhook in Langfuse

Go to Prompts > Webhooks in your Langfuse project
Click Create Webhook
(optional) filter events: filter by which prompt version events to receive webhooks (default: created, updated, deleted)
Set endpoint URL: https://<your-domain>/webhook/prompt
Save and copy the Signing Secret

Note: Your endpoint must return 2xx status codes. Langfuse retries failed webhooks with exponential backoff.

Sample Webhook Payload

Sample webhook payload:

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "timestamp": "2024-07-10T10:30:00Z",
  "type": "prompt-version",
  "action": "created",
  "prompt": {
    "id": "prompt_abc123",
    "name": "movie-critic",
    "version": 3,
    "projectId": "xyz789",
    "labels": ["production", "latest"],
    "prompt": "As a {{criticLevel}} movie critic, rate {{movie}} out of 10.",
    "type": "text",
    "config": { "...": "..." },
    "commitMessage": "Improved critic persona",
    "tags": ["entertainment"],
    "createdAt": "2024-07-10T10:30:00Z",
    "updatedAt": "2024-07-10T10:30:00Z"
  }
}

Step 2: Prepare Your GitHub Repo and Token for Sync

Create a .env file with your GitHub credentials:

GITHUB_TOKEN=<your_github_pat_here>
GITHUB_REPO_OWNER=<github_username_or_org>
GITHUB_REPO_NAME=<repo_name>
# (Optional) GITHUB_FILE_PATH=langfuse_prompt.json
# (Optional) GITHUB_BRANCH=main
# (Optional) REQUIRED_LABEL=production

Replace placeholders with your actual values. The server will commit prompts to langfuse_prompt.json on the main branch by default. If REQUIRED_LABEL is set, only prompts with that specific label will be synced to GitHub.

GitHub PAT Permissions for Sync

For the webhook to work, your GitHub Personal Access Token needs minimal permissions:

Permission Type	Required Permissions
Required Permissions	Contents: Read and write, Metadata: Read-only
Legacy Token Scopes	For public repositories: `public_repo` scope, For private repositories: `repo` scope

Step 3: Implement the FastAPI Webhook Server

Create main.py with this FastAPI server:

from typing import Any, Dict
from uuid import UUID
import json
import base64

import httpx
from pydantic import BaseModel, Field
from pydantic_settings import BaseSettings, SettingsConfigDict
from fastapi import FastAPI, HTTPException, Body

class GitHubSettings(BaseSettings):
    """GitHub repository configuration."""
    GITHUB_TOKEN: str
    GITHUB_REPO_OWNER: str
    GITHUB_REPO_NAME: str
    GITHUB_FILE_PATH: str = "langfuse_prompt.json"
    GITHUB_BRANCH: str = "main"
    REQUIRED_LABEL: str = ""  # Optional: only sync prompts with this label

model_config = SettingsConfigDict(
        env_file=".env",
        env_file_encoding="utf-8",
        case_sensitive=True
    )

config = GitHubSettings()

class LangfuseEvent(BaseModel):
    """Langfuse webhook event structure."""
    id: UUID = Field(description="Event identifier")
    timestamp: str = Field(description="Event timestamp")
    type: str = Field(description="Event type")
    action: str = Field(description="Performed action")
    prompt: Dict[str, Any] = Field(description="Prompt content")

a...

Dependencies

Install dependencies:

pip install fastapi uvicorn pydantic-settings httpx

Running Locally

Run locally:

uvicorn main:app --reload --port 8000

Test the health endpoint at http://localhost:8000/health. Use ngrok or similar to expose localhost for webhook testing.

Step 4: Deploy and Connect the Server

Deploy: Use Render, Fly.io, Heroku, or similar. Set environment variables and ensure HTTPS is enabled.
Update Webhook: In Langfuse, edit your webhook and set the URL to https://your-domain.com/webhook/prompt.
Test: Update a prompt in Langfuse and verify a new commit appears in your GitHub repository.

Security Considerations

Verify signatures: Use the signing secret and x-langfuse-signature header to validate requests
Limit PAT scope: Use fine-grained tokens restricted to specific repositories
Handle retries: The implementation is idempotent - duplicate events won't create conflicting commits