AI automation pipelines
Two unattended pipelines. One is live and publishing; one is retired and kept here as the earlier design. Both turn scraped source material into published output with a schedule, a quality gate and a way to notice when they stop.
The Kopi Log
Problem
I wanted a site that publishes original tech-news analysis on a schedule without me in the loop, and that I could still trust. The hard part is not the writing. It is what happens when the LLM backend is down, a dependency drifts and the CI gate goes red, or a scheduled run is missed and nobody notices for days.
How source becomes output
The orchestrator is idempotent: articles are marked seen only after their post is published, so a still-developing story is reconsidered on the next run.
Orchestration and scheduling
A Kubernetes CronJob runs the pipeline Monday, Wednesday and Friday at 18:00 Singapore time. The repo is both the codebase and the content store, so a published post is a reviewable commit and git log --author=kopilog-bot lists every run that ever shipped. Because the CronJob pulls the pipeline image that CI builds from the same repo, merging a pipeline change deploys it before the next scheduled run: the system ships itself.
Deployment manifests live in the GitOps repo (CronJob, web Deployment, image-automation policy, SOPS-encrypted secrets). A small shared volume holds the only runtime state: the dedup database, the image cache and the cover images, which the web tier mounts read-only.
Failure handling and observability
- Decoupled content deploy
The web image build has no dependency on the Python quality gate. A dependency drift once turned a red gate into a week of frozen content. A red gate still fails the pipeline loudly (webhook → Telegram); it cannot hold posts hostage. - Self-healing freshness probe
A cluster-side probe checks every 30 minutes whether a publish slot was missed. Once the LLM backend is healthy again it creates a catch-up job. Its RBAC can create Jobs and nothing else. Alerting is deliberately narrow: one green ping when it heals, red only when a deploy is stuck or the site is unreachable. - No LLM credentials in the cluster
The pipeline calls a small HTTP proxy on a separate host that runs the subscription CLI. The adapter has retry, backoff and defensive JSON extraction; the whole test suite runs with the network mocked.
Evidence
- Live
289 posts listed on the home page, checked at build time on 2026-11-04. Unattended since 2026-06-18; every post is a bot commit in the public repo. - Public source
github.com/ChristianKlass/kopilog: pipeline, tests, CI config, the freshness probe's logic in the README. Bot-authored commits are the audit trail. - Metrics not claimed
No traffic or reader numbers: none are collected.
Absurd Vault Historical
What it did
A pipeline that discovered high-scoring Reddit stories, rewrote them into 60 to 120 second scripts, generated a voiceover, produced word-level karaoke captions, assembled a 9:16 video with a music bed, generated titles and descriptions, and uploaded to YouTube Shorts with a shorter TikTok teaser. It ran from cron on a Proxmox container: one processing run a day, three upload slots a day, a daily optimiser that unlisted dead videos and checked titles for banned terms, and a weekly analysis pass.
Each stage read from the previous stage's work directory and wrote to its own; the pipeline was idempotent and each stage logged, skipped and continued on error. Health checks and an upload-recovery script covered the two failure modes that actually happened: expired OAuth tokens and half-finished uploads.
Why it stopped
Retired when the local GPU host that ran its model backends was decommissioned. Earlier, I let an automated bulk rename of published titles run without a review step; YouTube's recommendations dropped sharply and never recovered. The review gates and publish-through-git design in the later pipelines come from that.
Source is private and the channel is not linked. Shown here as the earlier design and the lesson, not as a result.