LLMOps on
Google Cloud Platform

Mark Edmondson

Agenda

  1. The Sunholo Team
  2. Define LLMOps
  3. Enabling LLMOps on GCP

The Sunholo Team

Mark Edmondson - Founder

  • Founder of Sunholo ApS from Nov 1st, 2023
  • Google Developer Expert - Google Cloud since 2015
  • MSci Physics, Kings College London
  • Wrote an O’Reilly book on Google Analytics 4 and Google Cloud integrations

code.markedmondson.me

Electric Sheep - Company Brain

  • An LLM bot, prototyped in the blog post.
  • Evolved into main executor agent
  • Infinite memory
  • Langchain Retrieval Augmented Generation (RAG) bot

Conversations with a bot

Voight-Kampff - Junior Developer

  • Writes and executes code based on prompts
  • Uses same GCP infrastructure as Electric Sheep
  • Interacts with other bots
  • openinterpreter.com bot

Watching a bot code

An army of bots

Sunholo aims to be a post-LLM company

  • Custom bots for each business function
  • Agents running in private secure environments
  • Private data mainly interacted with via LLMs

Define LLMOps

LLMOps for Mark means…

What tech does LLMOps use?

LLMOps builds on top of DataOps and MLOps, but with extra requirements such as…

  1. Parsing inputs
  2. Cognition (MLOps can help here)
  3. Document store (DataOps can help here)
  4. Vectorstore embeddings

  1. Parsing input
    • LLM rephrasing
    • Image/Text/Audio
    • Prompt engineering

1. Parsing input

  1. The model
    • Cognition
    • Tailoring size of model to task
    • Finetuning (MLOps)

2. Cognition

  1. Document store
    • Source of truth
    • Data pipelines (DataOps)
    • Structured data (LLMs writing SQL?)
    • Unstructured data

3. Document store

  1. Vectorstore embeddings
    • A new datatype for most companies
    • New uses beyond LLMs
    • Embedding type
    • Chunking
    • Parsing of documents

4. Vectorstore

Enabling LLMOps on GCP

Open source LLM Agents

  • Langchain - modular LLM flows
  • LlamaIndex - advanced RAG
  • LiteLLM - proxy to standardise interacting with all LLMs, local and API based
  • Unstrucutured - easy parsing of documents to chunks
  • Autogen - Multiple agents talking to one another
  • OpenInterpreter - Agent executing its own code

LLMOps for Electric Sheep

  • Retrieval augmented generation (RAG)
  • Documentation is the new oil
  • All Sunholo documents, git repos, emails, notes, conversations, R&D etc.

Langchain ConversationalRetrievalChain

LLMOps for Voight-Kampff

  • Using LLMs to create code and scripts it then executes in a virtual environment
  • Non-interactive mode
  • Pick LLM to run locally or via API

Voight-Kampff and post-LLM software engineering

  • Executing Code within Docker containers
  • Terraform IaC gives agents superpowers
  • Best practices of GitOps/CI/CD/Testing/Documentation all enable agents

Voight-Kampff Triggers

  • Triggers:
    • CI/CD alerts to prompt agent fixing code
    • Scheduled Code development and refactoring
    • GitHub issue triage
  • It will build itself, the more systems are in code

Summary

  • This is just the beginning of an LLM revolution
  • post-LLM companies will use multiple agents
  • LLMOps builds on top of DevOps and MLOps
  • Sunholo offers LLMOps for GCP offering to accelerate your own use cases

Thanks

  • Updates soon at sunholo.com
  • mark@sunholo.com
  • linkedin.com/company/sunholo/

Sunholo - LLMOps on Google Cloud Platform - Nov 2023