Skip to content

oauth-token-factory-spec.md

Latest commit

 

History

History
230 lines (196 loc) · 13.3 KB

oauth-token-factory-spec.md

File metadata and controls

230 lines (196 loc) · 13.3 KB
Raw

OAuth token factory in Dapier

Outcome

Dapier owns named OAuth connections for multiple providers and accounts. An operator signs in with the existing DataTalksClub (DTC) shared identity, grants provider consent in Dapier, and assigns connections to agents. A CLI uses the same DTC identity to obtain a usable, short-lived provider access token for an authorized agent. Provider refresh tokens and client secrets stay in Dapier.

Initial connections:

Connection ID Provider account Minimum scope Consumer
youtube-datatalksclub DataTalksClub channel UCDvErgK0j5ur3aLgn6U-LqQ https://www.googleapis.com/auth/youtube for existing publishing tools Zoomcamp publishing agents
youtube-personal @stolzenable (resolve and bind its channel ID during connection) https://www.googleapis.com/auth/youtube.upload AI Shipping Labs buildcamp upload agent (website #1680)

These are distinct connection records and distinct secret versions, even if they use the same Google OAuth client. Selecting a connection is mandatory for any token request or write operation. No implicit "current YouTube account". Additional providers (for example Dropbox) use the same lifecycle and access contract, with provider-specific adapters for consent parameters, refresh, identity verification, and token response details.

Existing Dapier foundation

  • src/admin.py already implements DTC OIDC browser login through AUTH_BASE_URL/AUTH_ISSUER, validates ID tokens, and issues a Dapier session.
  • The same module has OAuth start/callback handlers, provider endpoints for YouTube and Dropbox, connection metadata in CONNECTIONS_TABLE, and dapier/oauth/<connection_id> secrets in Secrets Manager.
  • src/web/ already has a Connections view. template.yaml deploys the admin API and callback in the sandbox account.

Extend these components. Do not create a second auth system, separate public OAuth callback Lambda, or a new standalone token-service repository. Separate modules inside Dapier are welcome so src/admin.py does not grow indefinitely.

Identity and authorization

  1. Human operators use DTC shared auth. Keep the existing web OIDC flow. Add an explicit operator authorization check after authentication; a valid DTC account alone does not grant credential administration.
  2. The CLI uses the same DTC issuer. Default: device pairing — the CLI shows a code, the operator approves it on the dapier /device page after DTC sign-in, and the API issues a revocable, rotating device session that authenticates as the operator's stable subject (Cognito has no native RFC 8628 grant). Alternative: authorization-code + PKCE with a loopback callback and a dedicated public/native CLI client registered in the shared-auth stack. Neither flow copies a web session cookie or stores a DTC password in the CLI. The API verifies the operator's subject and expiry on every call.
  3. Store connection grants keyed by stable DTC subject and agent identity, with explicit connection ID and allowed operation (use, connect, admin). Deny by default. A caller cannot name an arbitrary agent to gain its grants.
  4. For a local agent running under the operator's CLI session, allow a short-lived, connection-scoped delegation. A headless agent must use a separately enrolled machine identity and its own grants; never ship the operator's refresh credential to a server. Specify enrollment, revocation, and expiry before enabling headless access.
  5. Dapier records who connected, used, refreshed, rotated, and revoked a connection. Audit records contain IDs and outcomes, never tokens, auth codes, client secrets, or full authorization URLs.

Connection and token model

Connection metadata in DynamoDB includes connection_id, provider, display name, owner DTC subject, requested/granted scopes, expected and verified provider account ID, status, secret ARN, timestamps, and a version. The provider account ID is immutable after first verified consent unless an operator explicitly replaces the connection. In particular, binding youtube-personal to the DataTalksClub channel must fail.

Secrets Manager stores the OAuth client credential, refresh token, and provider-specific durable metadata under one versioned secret per connection. Short-lived access tokens are renewed as needed; they do not need separate durable backups. DynamoDB does not contain bearer credentials. Preserve a valid refresh token when a refresh response omits refresh_token. Make connection edits and token refreshes safe under concurrent use: do not overwrite a newer secret with stale state, and do not lose a rotated refresh token. If refresh omits scope, preserve the previously granted scope; an explicit scope response, including a narrower or empty value, remains authoritative. Legacy missing scope metadata is repaired through a verified refresh using recorded consent as the fallback. Generic HTTP actions use this same refresh and identity-check path for OAuth connections. Redact provider error bodies before logging or returning them.

Provider adapters define authorization URL, token URL, extra consent fields, scope normalization, refresh method, account identity check, and revocation if supported. OAuth authorization-code flows use PKCE S256, exact state matching, fixed registered redirect URIs, bounded state lifetime, and single-use callbacks. Validate the callback's connection ID and operator identity. Never derive a redirect URI from an untrusted forwarded-host header.

Implemented deviation: the OAuth client ID/secret pair is shared per provider and deploys with the stack (CloudFormation parameters on the dapier stack, rendered into function environment variables), so operators never paste client credentials per connection; a connection may still carry explicit client credentials in its secret record when an imported refresh token was issued by a different client.

Token providers (Slack): a provider may skip OAuth entirely and authenticate with a directly supplied token. Slack connections accept a bot (xoxb-), user (xoxp-), or app-level (xapp-) token in the create/edit request; the token is verified against Slack auth.test (workspace ID and name become the verified account binding) and stored in the credentials table under the connection's credential_id. Scopes are not requested — they ride on the token. Editing the connection without a new token re-verifies and keeps the stored one; oauth/start and the agent connect flow reject token providers with a clear message. Workflow slack actions resolve connection_id through the connections table, falling back to a raw credential_id for global credentials.

For YouTube, verify the authenticated channel ID with channels.list(mine=true) before marking a connection ready or returning a token. If the upload-only scope cannot make that check, fail closed and document the minimum extra scope or a separate verification flow; do not silently widen youtube-personal to full read/write access. Verify the chosen Brand Account in the consent flow. The CLI's status output shows connection ID, provider, verified channel title/ID, scopes, and health, but no credentials.

CLI and API contract

Provide a dapier CLI (packaged from this repo) with commands along these lines; exact flags may follow the project's CLI conventions:

dapier auth login
dapier auth status
dapier auth logout
dapier connections list
dapier connections show youtube-personal
dapier connections connect youtube-personal
python3 gpull.py --as "youtube buildcamp" --agent buildcamp-uploader whoami

Scripts fetch a fresh provider access token in-script (see the google-data-pull skill's dtoken.py helper) and keep it in a local variable, never printed or persisted. Access tokens should have the provider's normal short lifetime. The flow never returns refresh tokens or client secrets. Scripts must not place credentials in command-line arguments, shell history, debug output, an .env file, or the deployed AI Shipping Labs website.

The API authenticates the CLI using DTC-issued identity, checks the connection grant, refreshes the provider token when necessary, and returns only the access token, expiry, scope, and verified provider account ID over TLS with Cache-Control: no-store. The CLI checks the returned account ID against the requested connection before handing the token to a process. The token endpoint is rate-limited and audited. Browser admin endpoints keep CSRF protection and an explicit operator authorization check.

The SDK or CLI integration for an agent must require a connection ID and an expected provider account ID. A publishing action fails before any upload if the returned account ID does not match. The existing youtube-manager-agent/.youtube/token.json remains the DataTalksClub local credential until its tools are explicitly migrated; do not replace it with the personal token. A compatibility adapter may materialize a Google authorized user JSON file for local tools, but must keep the per-connection separation and private-file rules.

Storage, deployment, and migration

Use Dapier's existing Lambda/API Gateway deployment, DynamoDB table, and Secrets Manager boundary. Narrow Lambda IAM from the current wildcard dapier/oauth/* management permission where practical, and grant CLI callers access through the Dapier API rather than direct Secrets Manager reads. Do not put secret values in CloudFormation, Git, Terraform state, or CI output.

The currently working DataTalksClub OAuth credential is in youtube-manager-agent/.youtube/ and backed up in the main account's zoomcamp-ops/youtube/{client-secret,token} secrets. Import it into Dapier as youtube-datatalksclub through a one-time operator command that transfers bytes without logging them, verifies the DataTalksClub channel ID, and checks that refresh works. Keep the existing backup and Hetzner copy until the new connection and consumers have been verified. Do not change or delete either backup as part of initial deployment.

Create youtube-personal through a fresh consent selecting @stolzenable. Do not copy the DataTalksClub token or assume that the Google account picker selected the intended channel. Nothing in the AI Shipping Labs production website needs this credential; the buildcamp migration is an operator job. There is an interim local upload-only cache in youtube-manager-agent/.youtube/personal/token.json; its channel identity has not been verified because youtube.upload cannot call channels.list. Treat it as untrusted input for migration, and complete verified consent in Dapier.

Do not automate the 107-video upload as part of this work. Confirm the current YouTube API upload quota and the unaudited-project private-video restriction before choosing an API uploader. Pilot one unlisted video and its gated embed as called for in #1680.

Acceptance checks

  1. Two YouTube connections can coexist; list/status makes their provider identities and scopes distinguishable without exposing secrets.
  2. DTC shared-auth login succeeds in browser and CLI; unauthenticated users, ordinary DTC users, expired sessions, and agents without a grant cannot request a token or administer a connection.
  3. Connecting youtube-personal with the DataTalksClub account, or vice versa, is rejected before the connection is marked ready. Reconnect and token refresh cannot silently change the bound provider account.
  4. An in-script fetch for one granted connection returns a fresh access token only to that script; requests for a different connection are denied. Token values do not appear in logs, CLI status, exceptions, or process arguments.
  5. Expired access tokens refresh, refresh-token rotation is preserved, and concurrent requests cannot regress the stored secret version. Revocation makes subsequent token requests fail.
  6. Callback tests cover state replay, wrong operator, changed host, bad code, missing/changed scopes, and provider errors. IAM/API tests cover main and sandbox deployment identities and connection-specific grants.
  7. The DTC credential import is verified against channel UCDvErgK0j5ur3aLgn6U-LqQ; existing YouTube manager and Zoomcamp flows continue to work. No personal token is added to their default paths.

Delivery order

  1. Extract Dapier's existing OAuth code into tested auth, provider, connection, and secret-store modules; preserve current web console behavior.
  2. Add channel binding, refresh lifecycle, per-connection authorization, and migration/import tooling.
  3. Add the DTC-authenticated CLI and token delivery contract.
  4. Enroll the two YouTube connections, verify end-to-end access, then migrate individual consumers deliberately.

Workflow worker refresh permissions

The workflow worker reads provider credentials and saves refreshed OAuth token records using version-conditional writes. Its dynamodb:PutItem permission is scoped to the credentials table and oauth#* partition keys; it cannot write operator credentials or delete credentials. This permits unattended workflows (such as Dropbox attachment uploads) to continue after access tokens expire. The template regression test protects this permission separately from token lifecycle tests, which do not enforce deployed IAM policies.