Skills Engineering CoreWeave GPU Workload Event Monitoring

CoreWeave GPU Workload Event Monitoring

v20260423
coreweave-webhooks-events
A comprehensive solution for monitoring CoreWeave cluster events and GPU workload status using webhooks. It tracks critical lifecycle events such as pod readiness, job completion, volume attachment, and node health. Essential for building automated scaling, alerting, and recovery pipelines for large-scale GPU-accelerated inference and training workloads.
Get Skill
416 downloads
Overview

CoreWeave Webhooks & Events

Overview

CoreWeave emits Kubernetes-native events and custom status callbacks for GPU workload lifecycle management. Monitor instance readiness, job completion, volume attachment, and node health to build automated scaling, alerting, and recovery pipelines for GPU-accelerated inference and training workloads.

Webhook Registration

const response = await fetch("https://api.coreweave.com/v1/webhooks", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.COREWEAVE_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    url: "https://yourapp.com/webhooks/coreweave",
    events: ["instance.ready", "job.completed", "volume.attached", "node.unhealthy"],
    secret: process.env.COREWEAVE_WEBHOOK_SECRET,
  }),
});

Signature Verification

import crypto from "crypto";
import { Request, Response, NextFunction } from "express";

function verifyCoreWeaveSignature(req: Request, res: Response, next: NextFunction) {
  const signature = req.headers["x-coreweave-signature"] as string;
  const timestamp = req.headers["x-coreweave-timestamp"] as string;
  const payload = `${timestamp}.${req.body.toString()}`;
  const expected = crypto.createHmac("sha256", process.env.COREWEAVE_WEBHOOK_SECRET!)
    .update(payload).digest("hex");
  if (!crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expected))) {
    return res.status(401).json({ error: "Invalid signature" });
  }
  next();
}

Event Handler

import express from "express";
const app = express();

app.post("/webhooks/coreweave", express.raw({ type: "application/json" }), verifyCoreWeaveSignature, (req, res) => {
  const event = JSON.parse(req.body.toString());
  res.status(200).json({ received: true });

  switch (event.type) {
    case "instance.ready":
      registerEndpoint(event.data.instance_id, event.data.gpu_type); break;
    case "job.completed":
      collectArtifacts(event.data.job_id, event.data.output_path); break;
    case "volume.attached":
      mountStorage(event.data.volume_id, event.data.node_name); break;
    case "node.unhealthy":
      drainAndReschedule(event.data.node_id, event.data.reason); break;
  }
});

Event Types

EventPayload FieldsUse Case
instance.readyinstance_id, gpu_type, ip_addressRegister inference endpoint
job.completedjob_id, output_path, duration_secondsCollect training artifacts
volume.attachedvolume_id, node_name, mount_pathConfirm storage availability
node.unhealthynode_id, reason, gpu_countDrain node and reschedule pods
instance.terminatedinstance_id, exit_code, gpu_typeClean up resources and alert

Retry & Idempotency

const processed = new Set<string>();

async function handleIdempotent(event: { id: string; type: string; data: any }) {
  if (processed.has(event.id)) return;
  await routeEvent(event);
  processed.add(event.id);
  if (processed.size > 10_000) {
    const entries = Array.from(processed);
    entries.slice(0, entries.length - 10_000).forEach((id) => processed.delete(id));
  }
}

Error Handling

IssueCauseFix
Signature mismatchClock skew between clustersValidate timestamp within 5-minute window
Duplicate instance.readyRescheduled pod on same nodeTrack instance IDs for deduplication
Stale node.unhealthyTransient GPU memory errorWait for consecutive events before draining
Missing output_pathJob failed before writingCheck exit_code before collecting artifacts

Resources

Next Steps

See coreweave-security-basics.

Info
Category Engineering
Name coreweave-webhooks-events
Version v20260423
Size 4.32KB
Updated At 2026-04-28
Language