# Covalent Launch: The GPU Cheatcode: Fine-tune 20 Llama Models in 5 Minutes

Santosh Radha, Agnostiq (Covalent)

AI Engineer World's Fair 2024

[Talk page](https://ai.engineer/talks/zHYQZFy0UVk-covalent-launch-gpu-cheatcode-fine-tune-20) · [Video](https://www.youtube.com/watch?v=zHYQZFy0UVk)

## Covalent Launch: The GPU Cheatcode: Fine-tune 20 Llama Models in 5 Minutes

Covalent’s launch demo follows locally written Python through remote GPU execution, model evaluation, CPU-based selection, and deployment behind a text-generation endpoint.

Familiarity with Python functions, decorators, and the distinction between model training and inference will help you follow the workflow.

### Send locally written Python to remote compute

[0:13](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=13s)

How do you fine-tune a collection of models from Python without first managing Docker containers or a Kubernetes cluster? Santosh Radha opens with that problem. The launch title names twenty **Llama** models, but he calls twenty arbitrary and describes an ambition to support hundreds. The five minutes refer to his presentation budget; the demonstration does not establish a timed twenty-model training result. He points viewers to a QR code for the accompanying code and examples, then introduces [Covalent](https://github.com/AgnostiqHQ/covalent) as an open-source/open-core product that sends locally written Python to a chosen compute backend.

The starting point is a function in a local Jupyter notebook. Add a decorator specifying the resources it needs, then press Shift+Enter. Radha’s first illustrative request selects an H100, 36 GB of memory, and a maximum runtime of two days. The function executes on the remote GPU backend, and its result comes back to the notebook. The memory quantities here and later are stated resource requests; the talk does not establish that they describe GPU VRAM.

The execution destination depends on how Covalent is used:

| Mode | Where the code runs |
| --- | --- |
| Open source | Your own attached compute |
| Hosted cloud | Covalent’s GPU cluster |
| Bring your own infrastructure | Your cloud or on-premises systems, orchestrated through Covalent |

Radha describes hosted billing in terms of GPU execution time: a five-minute H100 run incurs five minutes of usage, while a ten-second run incurs ten seconds. The local programming interface stays the same idea across these arrangements: specify compute, send the function, receive its result.

![Diagram connecting laptop Python code through Covalent to a compute cluster, with open-source and cloud options.](https://ai.engineer/api/talk-essay-media/f0b703bdfc146a4813031a8cb8d6c353cd3395e1e90e9ba8ab6a262274da6c86)

Develop locally, run on GPUs remotely with Covalent.

### Function jobs and inference services

[1:53](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=113s)

A **job** is a single function with an execution destination. Its decorator describes the compute requirement; the backend runs the function and returns a Python object. Radha describes the charge as belonging to that function’s execution. This makes the function boundary both a programming boundary and a resource-allocation boundary.

An **inference service** adds a different lifecycle. Define an initializer, define a Python handler for an endpoint such as `/generate`, and call `cc.deploy` from the notebook. Instead of only receiving a function result, the developer receives an API endpoint. In the launch presentation, Radha describes that endpoint as supporting scale-to-zero and scaling as requests arrive. These are the service capabilities described in the recording, rather than guarantees established here for a current SDK version.

Radha then gives three examples of custom scaling policies:

- **Scheduled capacity:** increase to ten GPUs at nine o’clock each day.
- **Utilization trigger:** scale when GPU utilization reaches 80%.
- **Request trigger:** scale when the request count reaches 1,000.

He also describes configurable authentication. The developer supplies Python functions and service policy; the platform handles the infrastructure work without requiring the developer to write Docker or Kubernetes configuration.

![Run Inference slide with Python service and endpoint definitions above a curl request.](https://ai.engineer/api/talk-essay-media/f1dabf06f7546d493ca2b3cf5e70dbc77f2cacc8e074b9f0f2742528c2275096)

Python code for an inference service and a sample request.

### A broader set of Python workloads

[3:07](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=187s)

Before moving into the fine-tuning code, Radha points to the example library. His examples range from real-time time-series analysis and inverted transformers for forecasting—which he describes as state of the art—to large language model serving and an entire AI model foundry built with Python. Fine-tuning is the small demonstration of a broader capability: coordinating compute for several kinds of workloads through the same language used to implement them.

![Tutorial cards beside a QR code labeled Examples and Documentation.](https://ai.engineer/api/talk-essay-media/520e604cc198603013afb11afd2a628fc58545c052e6be971c0df1b67fd0c3a0)

Covalent’s example library spans multi-agent workflows and LLM benchmarking.

### Give each task the resources it needs

[3:51](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=231s)

The code demonstration begins with ordinary training and evaluation functions in a local Python package. A training task wraps the fine-tuning function: it accepts a model and data, then returns a fine-tuned model. Radha assigns that task 24 CPU cores, one H100 GPU, 48 GB of memory, and an eighteen-hour maximum runtime. The resource specification surrounds the training function rather than changing its model-and-data interface.

Once training finishes, an evaluation task accepts the trained model and measures its accuracy. The workflow then ranks the candidates and picks the best-performing model for deployment. The selection task runs on a CPU machine: there is no reason to reserve a GPU merely to sort model scores. **Resource placement follows the work each function performs.**

### Connect training to deployment

[4:51](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=291s)

The selected model becomes an inference service. An initialization decorator marks the service setup, and a `/generate` endpoint generates text and returns a prediction. This completes the deployment target before the demonstration connects the individual tasks into a workflow.

The workflow itself uses a familiar Python structure: loop over candidate models, fine-tune each one, evaluate the result, and collect model–accuracy pairs. Sort those pairs, then deploy the winner. The orchestration logic can be expressed with the task functions passed explicitly:

```python
def train_select_deploy(
    models, data, fine_tune_task, evaluate_task,
    select_task, deploy_task,
):
    scored_models = []
    for model in models:
        trained_model = fine_tune_task(model, data)
        accuracy = evaluate_task(trained_model)
        scored_models.append((trained_model, accuracy))

    best_model = select_task(scored_models)
    return deploy_task(best_model)


def select_best(scored_models):
    ranked = sorted(
        scored_models,
        key=lambda item: item[1],
        reverse=True,
    )
    return ranked[0][0]
```

Here, `select_task` represents the CPU task wrapping `select_best`; the other task arguments represent the training, evaluation, and deployment boundaries. The collected pairs preserve the association between each trained model and its score, so selection returns the model that should be deployed.

A single-line dispatch submits the workflow to Covalent and creates a job in its application. Each function runs on the device assigned to it. The frontend then lets Radha inspect an individual evaluation step and the machine allocated to that step, connecting the Python task boundary to an actual execution record.

### Inspect the runs behind the workflow

[5:49](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=349s)

Radha closes by reporting two runtime and cost examples from the application:

| Run | Reported device | Reported runtime | Reported cost |
| --- | --- | --- | --- |
| Evaluation | `L14` | Six minutes | $0.87 |
| Another model run; task unspecified | V100 | Six minutes | $0.11 |

The first device name is transcribed as `L14`; its accelerator identity remains unclear. The second run’s task is not specified, and the presentation does not supply the model sizes, datasets, accuracy definition, or billing conditions needed to compare the two costs. These are reported individual runs, not a controlled accelerator comparison.

The completed path is nevertheless concrete: locally written Python defines training, evaluation, selection, and deployment; each function receives its own compute assignment; and the selected model becomes a service. Radha ends with that training-to-deployment workflow completed without the developer managing Docker or Kubernetes.

## Resources

- [Covalent](https://github.com/AgnostiqHQ/covalent) — Open-source Python workflow orchestration with executor plugins for cloud and on-premises compute.
- [iTransformer: Inverted Transformers Are Effective for Time Series Forecasting](https://arxiv.org/abs/2310.06625) — An inverted Transformer architecture that applies attention across variates for multivariate time-series forecasting.

## Slides

### The GPU Cheatcode

The speaker plans to demonstrate fine-tuning models directly from Python; 20 is an arbitrary count. The QR code points to the talk, code, and more examples.

[0:27](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=27s) [Title slide for fine-tuning 20 LLM models in five minutes, with a QR code for the talk.](https://assets.ai.engineer/sites/aiecode2025/sha256/b7c5046a54dfe18b8cf6f2d636803c0e2cc275593e218f57ca6c8807cd648a67.webp)

The GPU Cheatcode
Fine-tune 20 LLM Models in 5 Minutes

Find the talk here

Covalent.
www.covalent.xyz

### Develop locally, run on GPUs remotely

Covalent sends a decorated local Python function to remote compute. The open-source option uses your own cluster; the cloud option runs it on Covalent’s GPU cluster.

[0:47](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=47s) [A laptop sends Python jobs through Covalent to a compute cluster, using open-source or cloud options.](https://assets.ai.engineer/sites/aiecode2025/sha256/67c10d3c5dd2c0497fe0ac04a868825acacd3855d19aaeb24261203af775a8d6.webp)

Develop locally, run on GPUs remotly
Train, fine-tune, and deploy production-ready models from local Python.
No Kubernetes, no Docker, no YAML scripts.

Laptop

gpu=cc.CloudExecutor(
    gpu_type="h100",
    memory="36GB",
    time_limit="2 days"
)

@ct.electron(executor=gpu)
def fine_tune(base, **kwargs):
    .....
    return model

Batch-Jobs/Inference/Workflow

Covalent
Open-Source
Cloud

Compute Cluster
AWS/GCP/Azure
Bring Your Own Compute
Or
Use Our Server-less Compute

Covalent
www.covalent.xyz

### Submit Single Jobs / Workflows

A CloudExecutor specifies 10 H100 GPUs, CPUs and memory. The decorator sends the fine-tuning function to that resource; the speaker says the job returns a Python object.

[1:55](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=115s) [Covalent slide showing a Python decorator and cloud executor for a fine-tuning job.](https://assets.ai.engineer/sites/aiecode2025/sha256/bf248d1d9298e39f1af307078f03c3172cbc0b93c2555195dd4186f903ace636.webp)

Submit Single Jobs / Workflows

Just add a decorator to your function and you can easily ship it to any remote machine!

import covalent_cloud as cc
import covalent as ct

resource = cc.CloudExecutor(env="huggingface",
                            gpu_type="h100",
                            num_gpu=10,
                            num_cpus=150,
                            memory="200GB")

@ct.electron(executor=resource)
def fine_tune_model(data):
    # training logic
    return model

Covalent.
www.covalent.xyz

### Run Inference

Covalent’s example initializes a model on four H100 GPUs and exposes a /generate endpoint. The speaker says deployment returns an API that scales with requests, including scaling to zero.

[2:12](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=132s) [Covalent slide showing Python code for an autoscaling GPU inference service and an example request.](https://assets.ai.engineer/sites/aiecode2025/sha256/fb13f38cb420fbbe9891640c078c3445fb8eb10f6a008a4ff74ae235680fca91.webp)

Run Inference

Make your function auto-scale on any GPU backend with decorators!

import covalent_cloud as cc
from vllm import LLM, SamplingParams

gpu_h100x4 = cc.CloudExecutor(env="vllm",
                            num_cpus=10,
                            num_gpus=4,
                            gpu_type="h100")

@cc.service(executor=gpu_h100x4,name="llama3b",auth=True)
def vllm_lamma(model):
    llm = LLM(model=model)
    return {"llm": llm}

@vllm_lamma.endpoint("/generate")
def generate(llm, prompt, num_tokens=1500):
    sampling_params = SamplingParams(max_tokens=num_tokens)
    return llm.generate(prompts, sampling_params)

curl -s -X POST \
    "https://fn.prod.covalent.xyz/166380ef3f7d37dbf2a468613/generate" \
    -d '{"prompt": "Once upon a time, there was a", "num_tokens": 10}'

Covalent.
www.covalent.xyz

### Examples and Documentation

Covalent’s tutorial gallery spans time-series forecasting, multi-agent applications and LLM benchmarking. A QR code points viewers to examples and documentation.

[3:27](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=207s) [Covalent slide with tutorial cards and a QR code for examples and documentation.](https://assets.ai.engineer/sites/aiecode2025/sha256/8958b2df2e312c70c21583f772289afb4df7fd5edef5150a02eae60909e515a2.webp)

Examples and Documentation

From multi-agent workflows to LLM benchmarking workflows

Featured tutorials:
Multi Agent LLMs — Generative AI/LLM
Deploying Llama 3 inference for text generation — Generative AI/LLM
Deploy Stable Diffusion Inference Service for Realistic Image Generation for Marketing — Generative AI/LLM

All Tutorials
Zero-Shot Time Series Forecasting with Lag-LLama — Cloud; Machine learning
Inverted-Transformers - Training and Deploying a Multivariate Time Series forecasting Model — Cloud; Machine learning
Benchmarking Large Language Models — Cloud; Generative AI/LLM
Building a Dynamic Multi-Agent Financial Analyst with Llama — Cloud; Generative AI/LLM
Serving a Lightweight Text-to-Image Model — Cloud; Generative AI/LLM
Serving a Llama-Based Chatbot Model — Cloud; Generative AI/LLM

Covalent.
www.covalent.xyz

Interface text:
Search

### Fine-tuning and evaluation tasks

The notebook wraps fine-tuning and evaluation functions as GPU tasks, then ranks their outputs on a CPU. The GPU executor specifies one H100, 24 CPUs, 48GB of memory, and an 18-hour limit.

[3:47](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=227s) [Python notebook defining cloud executors and tasks for fine-tuning, evaluation, and model ranking.](https://assets.ai.engineer/sites/aiecode2025/sha256/0d3544a75b3301be443199de82c5ebdbf3162ea01e8f4ab62ee146ee838e28b8.webp)

from training_model import fine_tune, evaluate

from transformers import (AutoModelForCausalLM, AutoTokenizer,
                          BitsAndBytesConfig, TrainingArguments, pipeline)

import covalent as ct

import covalent_cloud as cc

gpu=cc.CloudExecutor(env="lora-fine-tuning", num_cpus=24, num_gpus=1, gpu_type="h100", memory="48GB", time_limit="18 hours")

@ct.electron(executor=gpu)
def fine_tune_task(model,data):
    model=fine_tune(model,data)
    return model

@ct.electron(executor=gpu)
def evaluate_task(model,data):
    accuracy=evaluate(model,data)
    return accuracy

cpu=cc.CloudExecutor(env="lora-fine-tuning", num_cpus=24, memory="48GB", time_limit="18 hours")

@ct.electron(executor=cpu)
def rank_sort_models(eval_outputs):
    return sorted(eval_outputs, key=lambda x: x[1], reverse=True)

service_executor = cc.CloudExecutor(
    env="serving-env",

Interface text:
Code  File  Edit  Selection  View  Go  Run  Terminal  Window  Help
covalent-cloud-demo
fine_tune_demo.ipynb
Code  Markdown  Run All  Restart  Clear All Outputs  Variables  Outline
voice-clone (Python 3.10.13)
MagicPython
Launchpad  Live Share  LF  Cell 2 of 2  Indents: 1  3 Spell  Prettier  [off]

### Cloud executor configuration

The notebook assigns fine-tuning and evaluation to an H100 GPU executor, ranks models on a CPU executor, and configures a separate L40S GPU executor for serving.

[3:55](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=235s) [Python notebook showing fine-tuning and ranking tasks with a service executor configuration.](https://assets.ai.engineer/sites/aiecode2025/sha256/7c6da60d67a93a8fad792c59dc996e703ece68521890dd0addcc9a4d454c3cd8.webp)

import covalent as ct

import covalent_cloud as cc

gpu=cc.CloudExecutor(env="lora-fine-tuning", num_cpus=24, num_gpus=1, gpu_type="h100", memory="48GB", time_limit="18 hours")

@ct.electron(executor=gpu)
def fine_tune_task(model,data):
    model=fine_tune(model,data)
    return model

@ct.electron(executor=gpu)
def evaluate_task(model,data):
    accuracy=evaluate(model,data)
    return accuracy

cpu=cc.CloudExecutor(env="lora-fine-tuning", num_cpus=24, memory="48GB", time_limit="18 hours")

@ct.electron(executor=cpu)
def rank_sort_models(eval_outputs):
    return sorted(eval_outputs, key=lambda x: x[1], reverse=True)

service_executor = cc.CloudExecutor(
    env="serving-env",
    num_cpus=24,
    num_gpus=1,
    gpu_type="l40s",
    memory="48GB",
    time_limit="18 hours",
)

Interface text:
Code  File  Edit  Selection  View  Go  Run  Terminal  Window  Help
covalent-cloud-demo
fine_tune_demo.ipynb
Code  Markdown  Run All  Restart  Clear All Outputs  Variables  Outline
voice-clone (Python 3.10.13)
MagicPython
Launchpad  Live Share  LF  Cell 2 of 2  Indents: 1  3 Spell  Prettier  [off]

### Model service and generation endpoint

The notebook loads a model and tokenizer into a text-generation pipeline, then defines a /generate endpoint that returns the generated text. The presenter describes using it to serve the best model.

[4:44](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=284s) [Python notebook defining a model-serving executor, text-generation pipeline, and generation endpoint.](https://assets.ai.engineer/sites/aiecode2025/sha256/c5bff921b0394e3e50da44a5f0bb5fa4ab40c8084ddf4a04813e81e44d6d1c82.webp)

[illegible]xecutor=cpu)
def rank_sort_models(eval_outputs):
    return sorted(eval_outputs, key=lambda x: x[1], reverse=True)

service_executor = cc.CloudExecutor(
    env="serving-env",
    num_cpus=24,
    num_gpus=1,
    gpu_type="l40s",
    memory="48GB",
    time_limit="18 hours",
)

@cc.service(executor=service_executor)
def best_ranked_llm_service(model_path):
    model = AutoModelForCausalLM.from_pretrained(model_path)
    tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
    pipe = pipeline(task="text-generation", model=model, tokenizer=tokenizer)
    return {"pipe": pipe}

@best_ranked_llm_service.endpoint("/generate", name="Generate Response")
def generate(pipe=None, prompt=None,):
    output = pipe(prompt)
    gen_text = output[0]['generated_text']
    return gen_text

@ct.lattice
def workflow(models_to_finetune, data_train, data_test):
    ft_outputs = []

Interface text:
Code  File  Edit  Selection  View  Go  Run  Terminal  Window  Help
covalent-cloud-demo
fine_tune_demo.ipynb
Code  Markdown  Run All  Restart  Clear All Outputs  Variables  Outline
voice-clone (Python 3.10.13)
Launchpad  Live Share  Spaces: 4  LF  Cell 2 of 2  Indents: 0  3 Spell  Prettier  [off]

[4:55](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=295s) [Notebook code configures a model service and a text-generation endpoint.](https://assets.ai.engineer/sites/aiecode2025/sha256/f9f29f8c22de9dd93f22725a2c4de967a335d38f2fc0519d5909e7d36fa80132.webp)

service_executor = cc.CloudExecutor(
    env="serving-env",
    num_cpus=24,
    num_gpus=1,
    gpu_type="l40s",
    memory="48GB",
    time_limit="18 hours",
)

@cc.service(executor=service_executor)
def best_ranked_llm_service(model_path):
    model = AutoModelForCausalLM.from_pretrained(model_path)
    tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
    pipe = pipeline(task="text-generation", model=model, tokenizer=tokenizer)
    return {"pipe": pipe}

@best_ranked_llm_service.endpoint("/generate", name="Generate Response")
def generate(pipe=None, prompt=None,):
    output = pipe(prompt)
    gen_text = output[0]['generated_text']
    return gen_text

@ct.lattice
def workflow(models_to_finetune, data_train, data_test):
    ft_outputs = []

    for model in models_to_finetune:
        ft_model = fine_tune_task(model, data_train)
        accuracy = evaluate_task(ft_model, data_test)
        ft_outputs.append((ft_model, accuracy))

Interface text:
Code  File  Edit  Selection  View  Go  Run  Terminal  Window  Help
covalent-cloud-demo
fine_tune_demo.ipynb
Code  Markdown  Run All  Restart  Clear All Outputs  Variables  Outline
voice-clone (Python 3.10.13)
Launchpad  Live Share  Spaces: 4  LF  Cell 2 of 2  Indents: 1  3 Spell  Prettier  [off]

### Fine-tuning workflow

The workflow fine-tunes and evaluates each model, ranks the results by accuracy, and starts a generation service for the best-ranked model.

[5:03](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=303s) [Notebook code fine-tunes and evaluates models, ranks the results, and starts a service for the best model.](https://assets.ai.engineer/sites/aiecode2025/sha256/a4bc4796e20bfb9942a981f8745140c41bd54c0ce77f253bacb58e985afc7bf1.webp)

[illegible]="18 hours",
)

@cc.service(executor=service_executor)
def best_ranked_llm_service(model_path):
    model = AutoModelForCausalLM.from_pretrained(model_path)
    tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
    pipe = pipeline(task="text-generation", model=model, tokenizer=tokenizer)
    return {"pipe": pipe}

@best_ranked_llm_service.endpoint("/generate", name="Generate Response")
def generate(pipe=None, prompt=None,):
    output = pipe(prompt)
    gen_text = output[0]['generated_text']
    return gen_text

@ct.lattice
def workflow(models_to_finetune, data_train, data_test):
    ft_outputs = []

    for model in models_to_finetune:
        ft_model = fine_tune_task(model, data_train)
        accuracy = evaluate_task(ft_model, data_test)
        ft_outputs.append((ft_model, accuracy))

    ranked_ft_outputs = rank_sort_models(ft_outputs)
    model_client = best_ranked_llm_service(ranked_ft_outputs[0][0])

    return ranked_ft_outputs, model_client

Interface text:
Code  File  Edit  Selection  View  Go  Run  Terminal  Window  Help
AIEWF - Exhibition · 5 h 39 m left
Thu, Jun 27 11:21 AM
covalent-cloud-demo
fine_tune_demo.ipynb
Code  Markdown  Run All  Restart  Clear All Outputs  Variables  Outline
voice-clone (Python 3.10.13)
MagicPython
Launchpad  Live Share  Spaces: 4  LF  Cell 2 of 2  Indents: 1  3 Spell  Prettier  [off]

### Completed workflow graph

Completed model evaluations feed a ranking step and an electron list; the ranked output feeds a model service. The dispatch shows a 51m 32s runtime.

[5:42](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=342s) [Covalent displays a graph of completed model evaluations feeding ranking and model-serving steps.](https://assets.ai.engineer/sites/aiecode2025/sha256/4ec1444f9a60008a821766d2dcdb267986746effb54d96a9a6ca6f9d01cd8c36.webp)

Pending
5:08 am
27 Jun

Running
5:08 am
27 Jun

Completed
6:00 am
27 Jun

Cost
Total Electrons
51m 32s
Runtime

Results
Expand Code
[[(PosixPath('/volumes/lora-fine-tuning/Qwen1.5-1... [illegible]

Show more
Graph View
Detail View

Function Serve
Best Ranked Llm ...
11m 48s
Completed
$0.11

All Functions
Evaluate Model — 4m 11s
Evaluate Model — 4m 14s
Evaluate Model — 6m 38s
Evaluate Model — 29m 12s
Evaluate Model — 28m 19s
Evaluate Model — 14m 58s
Evaluate Model — 9m 6s
Evaluate Model — 6m 36s

Workflow graph:
Evaluate Model — 12, 15, 18, 21
Evaluate Model — 24, 27, 30, 33
View More
Evaluate Model — 36, 39, 42, 45
Evaluate Model — 48, 51, 54, 57
Rank Sort Models — 60
Electron List — 61
Best Ranked Llm Service — 62
Electron List — 65
Edge labels: arg[0], [illegible], arg[2], arg[3], arg[4], [illegible], arg[6], [illegible], arg[8], arg[9], arg[10], arg[11], arg[12], [illegible], arg[14], [illegible], arg[16], arg[17], arg[18], arg[19], eval_outputs, arg[0], arg[0], arg[1]

Interface text:
Covalent
Home
Dispatches
Function Serve
Shared
Env & Secrets
Chrome
File
Edit
View
History
Bookmarks
Profiles
Tab
Window
Help
Covalent
app.covalent.xyz/dispatches/defb37e0-ba7e-457b-ae1d-54257998cb52
Finish update
Thu, Jun 27 11:21 AM

### Evaluation details

Covalent’s detail view shows one completed model evaluation: it ran for 6m 40s on an L40 GPU and cost $0.87. The result includes a model path and a value of 0.936.

[5:46](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=346s) [Covalent shows a completed model evaluation with its result, runtime, cost, and machine specifications.](https://assets.ai.engineer/sites/aiecode2025/sha256/aaeef56f3b14d10dac8598c8860d94450a58a5b220915cfc663813d0ebdd3d1c.webp)

Pending
5:08 am
27 Jun
Running
5:08 am
27 Jun
Completed
6:00 am
27 Jun
51m 32s
Runtime
Results
[[(PosixPath('/volumes/lora-fine-tuning/Qwen1.5-1...
Show more

Graph View
Detail View

Function Serve
Best Ranked Llm ...
11m 48s
Completed
$0.11

All Functions
Evaluate Model  4m 11s
Evaluate Model  4m 14s
Evaluate Model  6m 38s
Evaluate Model  29m 12s
Evaluate Model  28m 19s
Evaluate Model  14m 58s
Evaluate Model  9m 6s
Evaluate Model  6m 36s

Node Id: 27
Function Name: Evaluate M...
Status
Completed
Cost
$0.87
Pending
5:08 am
27 Jun
Running
5:08 am
27 Jun
Completed
5:15 am
27 Jun
Runtime  6m 40s
Start Time ~ End Time  27 Jun, 05:08:45 - 27 Jun, 05:15:25

Result
Input
Code
Std Out
(PosixPath('/volumes/lora-fine-tuning/Phi-3-mini-128k-instruct_[illegible]'), 0.936)
Expand

Specifications
l40
GPU
CPU Shares  24
Memory  48
Time Limit  28800
GPUs  1
lora_fine_tu...
channels: 0
conda: 2
pip: 14

Interface text:
Chrome
File  Edit  View  History  Bookmarks  Profiles  Tab  Window  Help
AIEWF - Exhibition · 5h 39m left
Thu, Jun 27 11:21 AM
Covalent
app.covalent.xyz/dispatches/defb37e0-ba7e-457b-ae1d-54257998cb52
Finish update
Covalent
Home
Dispatches
Function Serve
Shared
Env & Secrets

## Full transcript

[0:00](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=0s) [on hold music]

[0:13](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=13s) So the talk is actually going to be about, um, uh, how you run things extremely easy directly from Python. And the example that I'm going to show you here is obviously I just have five minutes in on my end, but I'm going to try my best to showcase how you can fine-tune pretty much, twenty is an arbitrary

[0:29](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=29s) number here, but hundreds of models that you can do right from Python without needing anything like Kubernetes, Docker, or anything on your side in. Uh, so before that, you can find the talk and the actual code for what I'm gonna do in this QR code, and you'll find a lot more interesting examples over there to try out

[0:45](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=45s) and run as well. Um, okay. So what do we do? Um, so Covalent is an open source/open core, uh, product on its end. And what we do is we help people write Python locally and ship the code to any kind of compute backend that you need to send it to.

[1:03](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=63s) So what that means is, hey, you have a Python function that you want to run on a GPU, um, in your local laptop, open up a Jupyter Notebook, add a single decorator on top to say, "Hey, I want to run this on H100 with thirty-six gigs of memory for two days maximum time limit," and press Shift+Enter in

[1:19](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=79s) your Jupyter Notebook, and that's it. The code gets shipped to a backend in a GPU, and you get back the result on your side in. In the open source case, it sends it to your own compute.

[1:28](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=88s) You can attach your own compute cluster, and it runs over there. In the cloud case, it runs in our GPU cluster, and you just pay for the GPU time that it runs in.

[1:36](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=96s) So it runs for five minutes, you pay for five minutes of H100. It runs for ten seconds, you pay for ten seconds of H100s on your side in. You can also bring your own compute and attach to us, and we'll help you orchestrate the entire compute that you're handling it on your side, be it your own cloud

[1:49](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=109s) or on-prem systems or whatever it is on your end in.

[1:53](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=113s) Okay. So, uh, Covalent basically has a bunch of parameters that you define in. You can submit in jobs, which are called single functions. So essentially all you need to do is, as I said, add a single decorator on top and say, what is the compute that you need to ship it to?

[2:07](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=127s) It goes there, it runs, and you get back the Python object back, and you just pay for the function that you are running in.

[2:13](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=133s) We also let you run inferences, and again, it's completely Pythonic. You don't Dockerize, you don't run Kubernetes cluster, you don't do anything. You just say, "Hey, I have an initializer function, and I have a-- I need an endpoint called /generate."

[2:27](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=147s) And you define your Python functions. You click a single cc.deploy command, uh, in your Jupyter Notebook. The entire service gets shipped to us, and we scale. You get back an API endpoint that scales to zero or scales in its requests as and when your new request comes in.

[2:41](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=161s) You can define your custom auto-scaling mechanism. Like, "Hey, I want to auto-scale it to ten GPUs exactly at nine o'clock every day," or, "I wanna auto-scale whenever my GPU utilization hits in eighty percent," or, "I wanna auto-scale whenever the number of requests I get in is thousand."

[2:56](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=176s) Uh, so you can define whatever auto-scaling you want. You can define authentication and everything, and everything happens in the background for you. You don't even touch a single code of Kubernetes or Docker or anything on your side in.

[3:07](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=187s) And the talk I'm gonna give in is a very tiny example, um, of what we do from our side. But if you go to this link in, there's a whole host of examples, uh, that you can run in right from real-time time series analysis to, uh, you know, using inverted transformers for time series, which is like a

[3:25](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=205s) state-of-the-art, uh, time series transformer on its end, uh, running in large systems, um, large language models on your serving, uh, systems, and even building an entire AI model foundry out of our just pure Pythonic code, uh, on your side in.

[3:40](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=220s) So without further ado, I'll quickly run through the code example of how you do essentially fine-tune a bunch of huge set of models, uh, directly just from Python on your end.

[3:51](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=231s) And I'll also show you how it looks like, uh, from the front-end side as well. So, um, it's rather simple. All you do is I have written a bunch of, uh, normal Pythonic training functions in my local package called fine-tune and evaluate on our end.

[4:06](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=246s) And what we are going to do is, hey, I'm going to define a Python task, which essentially calls in my fine-tune function, which is going to accept a model and data and return back a fine-tune model.

[4:15](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=255s) So this is a simple Python function, and I'm gonna say, hey, I want to run it on a twenty-four core CPU with one GPU in it of type H100 with forty-eight gigs of memory, and going to give a max limit of eighteen hours on it.

[4:27](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=267s) And then I'm gonna say, hey, I'm gonna-- Once the model is done, I'm gonna accept the model and then evaluate its accuracy on its end. And finally, I'm going to just sort the model among all the best models and then pick the best model in it.

[4:39](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=279s) And I want this to run on a CPU-based machine. I don't wanna waste GPU for my sorting or whatever I'm gonna do in on my end. And finally, I'm gonna deploy the best model that I figured, um, in its-- that has performed well on my end.

[4:51](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=291s) And this is like in a simple decorator to add to say, "Hey, this is my initialization service, and I'm gonna create an endpoint called /generate. Um, and I'm going to generate the text and give back the prediction."

[5:04](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=304s) And finally, this is where the magic happens. To tie together all of these things, what I do is I'm gonna create a workflow where I'm pretty much just going to simply loop over a bunch of models to fine-tune, call the fine-tune function, evaluate the task and get the accuracy, make a list of all the models and accuracy,

[5:20](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=320s) sort the best models, and then deploy the model from my end. And this is completely Pythonic. And once you dispatch this to our server, which is essentially calling a single line over here, what you will go back and see is a new job that creates in our application, and all of the functions that you called will run

[5:39](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=339s) in the respective devices that you just defined. So for instance, here is, uh, one of the evaluation step that ran in, and it has its own machine that we ran in.

[5:49](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=349s) It ran in L14. It ran for six minutes, and you get back just eighty-seven cents to evaluate your model in. Another model ran in V100 on its end, and it ran for six minutes again.

[6:02](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=362s) It costed eleven cents to do it in. And in total, you finally have deployed, tri-- fine-tuned, and trained completely in Python without needing anything like Docker or Kubernetes on your end.

[6:13](https://www.youtube.com/watch?v=zHYQZFy0UVk&t=373s) And we have a booth over there. Do visit us, and we can have more chat over there. Thank you, guys. [outro music]
