Web API Implementation Plan#

Overview#

Add a Huma +chi -based REST API to scafctl, started via a new scafctl serve command. The API mirrors all major CLI features (run, render, lint, eval, catalog, solutions, providers, config, etc.) with Entra OIDC authentication, Prometheus metrics, OpenTelemetry tracing, audit logging, CEL-based filtering, and admin endpoints.

Architecture follows the jqapi reference implementation: chi router → layered middleware → Huma endpoint registration → handler context with dependency injection. All operations are synchronous request-response initially, with an async task model planned as future work.

Phase 1: Server Foundation (pkg/api/)#

Step 1.1: Create API Server Package#

Create pkg/api/server.go with a Server struct modeled after jqapi’s internal/server/server.go.

Fields:

Constructor:

func NewServer(cfg *config.Config, opts ...ServerOption) (*Server, error)

Functional options pattern (like pkg/mcp/server.go):

• WithServerLogger(lgr logr.Logger)
• WithServerRegistry(reg *provider.Registry)
• WithServerAuthRegistry(reg *auth.Registry)
• WithServerVersion(version string)
• WithServerContext(ctx context.Context)

Methods:

• Router() *chi.Mux — returns root router for middleware setup
• SetAPIRouter(r chi.Router) — sets the API route group with heavy middleware
• APIRouter() chi.Router — returns API router (falls back to root if unset)
• API() huma.API — returns Huma API for endpoint registration
• InitAPI() — initializes Huma after middleware setup (must be called after SetupMiddleware)
• Start() error — starts the HTTP server (blocks until shutdown)
• Shutdown(ctx context.Context) error — graceful shutdown with configurable timeout

Graceful Shutdown:

• Signal handling: SIGINT/SIGTERM
• Context cancellation propagation
• Configurable shutdown timeout (default: 30s)
• Sets isShuttingDown flag so readiness probes return 503

Reference:

• jqapi internal/server/server.go — chi+Huma server pattern
• scafctl pkg/mcp/server.go — functional options, mergeContext for layered cancellation

Step 1.2: Create API Config Types#

Extend pkg/config/types.go by adding an APIServer field to the Config struct:

type Config struct {
    // ... existing fields ...
    APIServer APIServerConfig `json:"apiServer,omitempty" yaml:"apiServer,omitempty"`
}

New APIServerConfig struct:

type APIServerConfig struct {
    Host            string                    `json:"host"            yaml:"host"            doc:"Host to bind to"                                       example:"127.0.0.1"`
    Port            int                       `json:"port"            yaml:"port"            doc:"Port to listen on"                                     example:"8080"  maximum:"65535"`
    APIVersion      string                    `json:"apiVersion"      yaml:"apiVersion"      doc:"API version prefix"                                    example:"v1"`
    ShutdownTimeout string                    `json:"shutdownTimeout" yaml:"shutdownTimeout" doc:"Graceful shutdown timeout"                              example:"30s"`
    RequestTimeout  string                    `json:"requestTimeout"  yaml:"requestTimeout"  doc:"Default request timeout"                               example:"60s"`
    MaxRequestSize  int64                     `json:"maxRequestSize"  yaml:"maxRequestSize"  doc:"Maximum request body size in bytes"                    example:"10485760"`
    TLS             APITLSConfig              `json:"tls"             yaml:"tls"             doc:"TLS configuration"`
    CORS            APICORSConfig             `json:"cors"            yaml:"cors"            doc:"CORS configuration"`
    RateLimit       APIRateLimitConfig        `json:"rateLimit"       yaml:"rateLimit"       doc:"Rate limiting configuration"`
    Auth            APIAuthConfig             `json:"auth"            yaml:"auth"            doc:"Authentication configuration"`
    Compression     APICompressionConfig      `json:"compression"     yaml:"compression"     doc:"Response compression configuration"`
    OpenAPI         APIOpenAPIConfig           `json:"openAPI"         yaml:"openAPI"         doc:"OpenAPI specification configuration"`
    Profiler        APIProfilerConfig         `json:"profiler"        yaml:"profiler"        doc:"Profiler configuration"`
    Audit           APIAuditConfig            `json:"audit"           yaml:"audit"           doc:"Audit logging configuration"`
    Tracing         APITracingConfig          `json:"tracing"         yaml:"tracing"         doc:"OpenTelemetry tracing configuration"`
}

Sub-config structs:

Constants (add to pkg/settings/settings.go):

const (
    DefaultAPIPort            = 8080
    DefaultAPIHost            = "127.0.0.1"
    DefaultAPIVersion         = "v1"
    DefaultShutdownTimeout    = "30s"
    DefaultRequestTimeout     = "60s"
    DefaultMaxRequestSize     = 10 * 1024 * 1024 // 10MB
    DefaultCompressionLevel   = 6
)

Step 1.3: Huma API Initialization#

Create pkg/api/huma.go with Huma configuration builder methods:

func (s *Server) buildHumaConfig(apiVersion string) huma.Config
func (s *Server) configureOpenAPIInfo(humaConfig *huma.Config)
func (s *Server) configureOpenAPIServers(humaConfig *huma.Config)
func (s *Server) configureSecuritySchemes(humaConfig *huma.Config)
func (s *Server) configureDocsPaths(humaConfig *huma.Config, apiVersion string)

• Security scheme: OAuth2 with Entra client ID scope (api://{clientID}/.default)
• Uses humachi.New(apiRouter, humaConfig) adapter
• Configures docs path at /{version}/docs and OpenAPI at /{version}/openapi.json
• Default format: application/json

Reference: jqapi Server.buildHumaConfig(), Server.InitAPI()

Phase 2: Middleware Stack (pkg/api/middleware/)#

Step 2.1: Two-Layer Middleware Setup#

Create pkg/api/middleware.go:

func SetupMiddleware(router *chi.Mux, cfg *APIServerConfig) (chi.Router, error)

Returns the API route group (chi.Router) with heavy middleware applied. Health probes and operational endpoints registered on the root router bypass the API middleware entirely.

Global Middleware (all routes including health probes):

API Middleware (returned chi.Router, business endpoints only):

Security requirement: Returns error if Entra OIDC is enabled but misconfigured (refuses to start unauthenticated when auth is expected).

Reference: jqapi internal/api/middleware.go — exact same two-layer pattern

Step 2.2: Entra OIDC Auth Middleware#

Create pkg/api/middleware/auth.go:

• JWKS endpoint discovery and caching
• JWT validation: signature, expiration, audience, issuer
• Tenant ID and Client ID verification
• Extract claims and inject into request context (user identity, groups, roles)
• Leverage existing pkg/auth/entra/ for token handling patterns
• Follow jqapi’s middleware.NewAzureOIDCHandler() pattern

Can be developed in parallel with Step 2.3.

Step 2.3: Rate Limiting & Security Middleware#

Create pkg/api/middleware/ratelimit.go:

• Per-IP rate limiting using token bucket or sliding window algorithm
• Global rate limit configuration
• Per-endpoint rate limit overrides
• Returns 429 Too Many Requests with Retry-After header

Create pkg/api/middleware/security.go:

• Security headers: X-Content-Type-Options: nosniff, X-Frame-Options: DENY, Strict-Transport-Security (when TLS), Content-Security-Policy
• Request size limits: reject requests exceeding MaxRequestSize
• Max concurrent requests: chi Throttle-based in-flight request limiting

Can be developed in parallel with Step 2.2.

Step 2.4: Metrics Middleware#

Create pkg/api/middleware/metrics.go:

• Record per-request metrics: count, duration histogram, status code distribution
• Per-endpoint breakdown
• Integrate with existing pkg/metrics/ infrastructure

Step 2.5: Audit Logging Middleware#

Create pkg/api/middleware/audit.go:

Structured audit log for every API request including:

• Uses existing pkg/logger/ for structured output
• Configurable via APIServer.Audit.Enabled

Security: Body Redaction

Before writing request bodies to the audit log, sensitive fields are redacted to prevent credential leakage. The following patterns are redacted:

• Fields with keys matching password, secret, token, key, credential, authorization (case-insensitive)
• The Authorization HTTP header is never logged
• Redacted values are replaced with "[REDACTED]"

This prevents OWASP-listed sensitive data exposure through audit logs.

Step 2.6: OpenTelemetry Tracing Middleware#

Create pkg/api/middleware/tracing.go:

• Propagate trace context from incoming requests (W3C Trace Context headers)
• Integrate with existing pkg/telemetry/ OTLP pipeline
• Add span attributes: endpoint, method, status code, user identity
• Trace context flows through handler → domain package calls
• Create child spans for significant operations within handlers

Phase 3: Handler Context & Shared Types (pkg/api/)#

Step 3.1: Handler Context (Dependency Injection)#

Depends on Phase 1.

Create pkg/api/context.go:

type HandlerContext struct {
    Config           *config.Config
    ProviderRegistry *provider.Registry
    AuthRegistry     *auth.Registry
    Logger           logr.Logger
    IsShuttingDown   *int32
    StartTime        time.Time
}

func NewHandlerContext(
    cfg *config.Config,
    providerReg *provider.Registry,
    authReg *auth.Registry,
    logger logr.Logger,
    isShuttingDown *int32,
    startTime time.Time,
) *HandlerContext

All handlers access scafctl domain packages through this struct (like jqapi’s HandlerContext). Components can be swapped with mock implementations for testing.

Step 3.2: Error Handling#

Create pkg/api/errors.go:

func HandleError(ctx context.Context, err error, operation string, statusCode int, userMessage string) error
func HandleValidationError(ctx context.Context, fieldName, message string) error

• Log the error with context
• Map scafctl domain errors to HTTP status codes
• Return appropriate Huma errors
• Use RFC 7807 Problem Details format (application/problem+json)

Can be developed in parallel with Step 3.3.

Step 3.3: Shared Types#

Create pkg/api/types.go with request/response types. All types use Huma validation tags.

Pagination:

type PaginationInfo struct {
    Page       int  `json:"page"        minimum:"1"     maximum:"10000" example:"1"   doc:"Current page number (1-indexed)"`
    PerPage    int  `json:"per_page"    minimum:"1"     maximum:"1000"  example:"100" doc:"Items per page"`
    Total      int  `json:"total"       minimum:"0"     maximum:"1000000" example:"250" doc:"Total number of items"`
    TotalPages int  `json:"total_pages" minimum:"0"     maximum:"10000" example:"3"   doc:"Total number of pages"`
    HasNext    bool `json:"has_next"    example:"true"  doc:"Whether there is a next page"`
    HasPrev    bool `json:"has_prev"    example:"false" doc:"Whether there is a previous page"`
}

Health:

type HealthResponseBody struct {
    Status     string            `json:"status"     maxLength:"20" pattern:"..." example:"healthy" doc:"Health status"`
    Version    string            `json:"version"    maxLength:"50" pattern:"..." example:"v1.0.0"  doc:"Service version"`
    Uptime     string            `json:"uptime"     maxLength:"50" pattern:"..." example:"1h30m"   doc:"Service uptime"`
    Components []ComponentStatus `json:"components" maxItems:"100" doc:"Component health statuses"`
}

Root:

type RootResponse struct {
    Name        string    `json:"name"        maxLength:"100" example:"scafctl API" doc:"API name"`
    Version     string    `json:"version"     maxLength:"50"  example:"v1.0.0"      doc:"API version"`
    Description string    `json:"description" maxLength:"500"                        doc:"API description"`
    Links       []APILink `json:"links"       maxItems:"50"                          doc:"Related API links"`
}

Can be developed in parallel with Step 3.2.

Step 3.4: CEL Filtering#

Create pkg/api/filtering.go:

• Add filter query parameter to all list endpoints
• Reuse pkg/celexp/EvaluateExpression() to evaluate filter expressions against each result item
• Apply pagination after filtering

Security: CEL Sandboxing

User-supplied CEL filter expressions are evaluated in a restricted sandbox:

• Cost limit: All filter evaluations use celexp.WithCostLimit() (default: settings.DefaultAPIFilterCostLimit = 10000) to prevent denial-of-service via expensive expressions
• Max expression length: The filter query parameter is limited to 2000 characters via Huma validation (maxLength:"2000" on FilterParam)
• Read-only: CEL expressions cannot perform I/O, mutate state, or access the filesystem — enforced by the CEL runtime
• No custom functions: Filter expressions only have access to the standard CEL library plus the item variable

Example usage:

GET /v1/solutions?filter=name.startsWith("my-") && tags.exists(t, t == "production")
GET /v1/providers?filter=name == "helm" || name == "terraform"
GET /v1/catalogs?filter=solutions.size() > 10

Phase 4: API Endpoints (pkg/api/endpoints/)#

All steps in this phase depend on Phase 3 and can run in parallel with each other.

Step 4.1: Health & Operational Endpoints#

pkg/api/endpoints/health.go:

All registered on the root router (bypass auth/rate-limit middleware).

Step 4.2: Prometheus Metrics Endpoint#

Registered on the root router. Exposes:

• HTTP request count (by endpoint, method, status)
• Request duration histogram
• Active connections gauge
• Leverages existing pkg/metrics/ infrastructure

Step 4.3: Solution Endpoints#

pkg/api/endpoints/solutions.go:

Handlers delegate to existing domain packages: pkg/scaffold/, pkg/lint/, pkg/solution/, pkg/dryrun/.

Step 4.4: Provider Endpoints#

pkg/api/endpoints/providers.go:

Delegates to pkg/provider/ registry.

Step 4.5: CEL & Template Eval Endpoints#

pkg/api/endpoints/eval.go:

Delegates to pkg/celexp/ and pkg/gotmpl/.

Step 4.6: Catalog Endpoints#

pkg/api/endpoints/catalog.go:

Delegates to pkg/catalog/.

Step 4.7: Schema Endpoints#

pkg/api/endpoints/schema.go:

Delegates to pkg/schema/.

Step 4.8: Config & Settings Endpoints#

pkg/api/endpoints/config.go:

Delegates to pkg/config/ and pkg/settings/.

Step 4.9: Snapshot Endpoints#

pkg/api/endpoints/snapshot.go:

Delegates to pkg/dryrun/ and pkg/scaffold/.

Step 4.10: Explain & Diff Endpoints#

pkg/api/endpoints/explain.go:

Delegates to existing domain packages.

Step 4.11: Admin Endpoints#

pkg/api/endpoints/admin.go:

Authorization model:

• When Entra OIDC auth is enabled: admin endpoints require a valid JWT and the caller must have the admin role in their Entra token claims (roles claim). Requests without the admin role receive 403 Forbidden.
• When auth is disabled: admin endpoints are restricted to localhost-only requests. Requests from non-loopback IPs receive 403 Forbidden.

Registered under the “Admin” OpenAPI tag.

Step 4.12: Endpoint Registration#

Create pkg/api/register.go:

func RegisterEndpoints(api huma.API, router *chi.Mux, ctx *HandlerContext)
func RegisterEndpointsForExport(api huma.API) // for OpenAPI spec generation without starting server

• Central registration following jqapi pattern
• Each endpoint registered with huma.Register(api, huma.Operation{...}, handler)
• OpenAPI tags: Solutions, Providers, Eval, Catalogs, Schemas, Config, Health, Admin
• Security definitions per endpoint (auth-required vs public)

Phase 5: CLI Command & Wiring (pkg/cmd/scafctl/serve/)#

Step 5.1: scafctl serve Command#

Depends on Phases 1-4.

Create pkg/cmd/scafctl/serve/serve.go:

func CommandServe(cliParams *settings.Run, ioStreams *terminal.IOStreams, path string) *cobra.Command

Pattern follows pkg/cmd/scafctl/mcp/serve.go (dependency injection from Cobra context).

Flags:

runServe() flow:

1. Pull dependencies from Cobra context (logger, config, auth registry, provider registry)
2. Create server via api.NewServer(cfg, ...opts)
3. Setup middleware via api.SetupMiddleware(srv.Router(), cfg.APIServer)
4. Set API router: srv.SetAPIRouter(apiRouter)
5. Initialize Huma API: srv.InitAPI()
6. Register endpoints: api.RegisterEndpoints(srv.API(), srv.Router(), handlerCtx)
7. Register custom 404/405 handlers
8. Start server: srv.Start() (blocks until SIGINT/SIGTERM)

No business logic in this package — only dependency wiring.

Step 5.2: Wire into Root Command#

Edit pkg/cmd/scafctl/root.go:

• Add serve command to groupPlugin or a new groupServer command group
• Register: cmd.AddCommand(serve.CommandServe(cliParams, ioStreams, path))

Step 5.3: OpenAPI Export Subcommand#

Create pkg/cmd/scafctl/serve/openapi.go:

scafctl serve openapi --format json|yaml --output file.json

• Generates the full OpenAPI specification without starting the server
• Creates a minimal router + Huma API, registers all endpoints via RegisterEndpointsForExport()
• Follows jqapi’s exportOpenAPI pattern

Phase 6: MCP & Documentation#

Step 6.1: MCP Tools#

Can be developed in parallel with Step 6.2.

Create pkg/mcp/tools_api.go:

• s.registerAPITools() — tools for inspecting API server configuration, generating OpenAPI spec

Step 6.2: Documentation & Examples#

Can be developed in parallel with Step 6.1.

Phase 7: Testing#

Step 7.1: Unit Tests#

• *_test.go for every pkg/api/ file
• Middleware tests (like jqapi’s middleware_test.go)
• Audit logging tests (including body redaction verification), tracing tests, CEL filtering tests

Benchmark tests for every endpoint and middleware (like jqapi’s handlers_bench_test.go):

Step 7.2: Integration Tests#

• Add scafctl serve to tests/integration/cli_test.go
• Create tests/integration/api_test.go — API endpoint integration tests using httptest
• Solution integration tests in tests/integration/solutions/ for API-driven workflows

Step 7.3: Verify#

• task test:e2e

Files#

Existing Files to Modify#

New Files to Create#

Core server:

Middleware:

Endpoints:

CLI command:

Other:

Reference Patterns (existing code to follow)#

Verification#

Decisions#

Future Work (not in scope for this plan)#

1. Async Task Model#

For long-running operations (solution run/render/lint/test):

• POST /v1/solutions/run → returns 202 Accepted with a task ID
• GET /v1/tasks/{id} — poll for task status and result
• Optional SSE streaming (GET /v1/tasks/{id}/stream) for real-time progress events

Implementation:

• In-memory task store (sync.Map or mutex-protected map on Server struct)
• Task fields: ID, status (pending/running/completed/failed), progress %, log lines, result, timestamps
• Configurable TTL (e.g., 30 minutes), background goroutine evicts expired tasks
• Optional: filesystem persistence to XDG data dir for survival across restarts
• No external database required (no Redis, no SQLite, no Postgres)

2. Cursor-Based Pagination#

Add optional cursor-based pagination alongside the existing offset-based model for endpoints that may grow to large datasets. Return next_cursor in response metadata, accept cursor + limit query parameters. Useful if/when list endpoints are backed by external storage instead of in-memory registries.

3. Multi-Version API Coexistence#

Support running v1 and v2 simultaneously with different route groups and handler versions. Route groups are already version-prefixed to enable this.

4. WebSocket Support#

Persistent bidirectional connections for interactive solution development sessions.

Endpoint: GET /v1/ws upgrades to WebSocket (via gorilla/websocket or nhooyr.io/websocket).

Enables:

• Real-time streaming — solution run output (resolver outputs, provider progress, action logs) streamed as they happen, rather than waiting for the full HTTP response
• Interactive prompting — server pushes input prompts to client mid-execution and waits for responses over the same connection
• Live file watching — server watches solution files for changes and pushes change notifications to the client, enabling hot-reload development workflows
• Collaborative sessions — multiple clients connect to the same session and see each other’s changes

Complexity considerations:

• Connection lifecycle management (open, close, reconnect)
• Heartbeat/ping-pong for connection health
• Reconnection handling (client-side retry with backoff)
• Per-connection authentication (validate JWT on upgrade)
• State management (per-connection session state)

When to add: After the REST API is stable and a concrete client (web UI, VS Code extension) exists to consume it. chi supports WebSocket upgrades natively, so the existing router infrastructure carries over.