Twilio self-service phone numbers — runbook

⚠️ STATUS — 2026-04-25

The per-tenant subaccount model described below was abandoned on 2026-04-22 in commit 40499c3 (TWIN-79). All Twilio resources now live on the master account AC9022c2117ed47134c7f79f30c7fd8ea3; tenant isolation happens in our DB only. Reason: Twilio Regulatory Bundles reject subaccount purchases with error 21649, which broke self-service for every regulated number type (UK Local, UK Toll-Free, UK/PL Mobile).

What this means in practice:

• ensureSubaccount(_tenantId) always returns master credentials.
• getSubaccountClient(_tenantId) returns a master-scoped axios.
• New tenants do not get rows written to tenants.twilio_subaccount_*.
• resolveWebhookAuthToken(query) always falls through to the master TWILIO_AUTH_TOKEN.
• The endpoints, UI components, drift-sync cron and quota logic listed below all still apply — only the Twilio-side scoping changed.

The text below is preserved for historical context and to document code paths that still exist (some are dead-code candidates for a cleanup migration). TWIN-89 was closed as obsolete on 2026-04-25 confirming we are not reintroducing subaccounts.

Parent ticket: TWIN-40. Master plan: ../twin-40-master-plan.md.

TL;DR (historical — see status banner above)

Each Twinlix tenant has its own Twilio subaccount under our master account AC35d2276d8331bad91c8d7ee6bacf9b49. Numbers are purchased into that subaccount, webhooks are bound to it, and signatures are verified with the subaccount's own auth_token. The subaccount is provisioned lazily the first time the tenant hits any self-service endpoint.

Key pieces

Where	What
backend/src/services/twilio-subaccount.service.ts	ensureSubaccount, getSubaccountClient, releaseAllTenantNumbers
backend/src/api/routes/phone-numbers.ts	five self-service endpoints + legacy shared-pool list
backend/src/lib/crypto.ts	AES-256-GCM at-rest encryption for auth_token
backend/src/lib/crypto/twilio-verify.ts	resolveWebhookAuthToken + verifyTwilioSignature(signature, url, params, authToken)
backend/src/services/jobs/phone-numbers-drift-sync.job.ts	daily 03:15 reconciliation
backend/prisma/schema.prisma	Tenant.twilio_subaccount_*, extended PhoneNumber model
twinlix-admin-front/src/libs/DataSource/services/PhoneNumbersService.ts	frontend client for the 5+1 endpoints
twinlix-admin-front/src/components/1-ions/PhoneNumberList/	list component inside the voice-channel settings modal
twinlix-admin-front/src/components/1-ions/ConnectVoiceLineModal/	purchase modal
twinlix-admin-front/src/utils/twilioError.ts	error-code → user-copy mapping

Secrets

• Master Twilio credentials: TWILIO_ACCOUNT_SID + TWILIO_AUTH_TOKEN in the backend .env. Used to create subaccounts and to hit the global Pricing API. Current prod value points at account Twinlix (Full).
• Per-tenant subaccount credentials: stored in tenants.twilio_subaccount_sid (plain) and tenants.twilio_subaccount_auth_token_ciphertext (AES-GCM blob).
• AES key: SUBACCOUNT_ENCRYPTION_KEY in backend .env, base64-encoded 32-byte value. Generate with openssl rand -base64 32. Losing the key means every stored subaccount token becomes unreadable — document any rotation in a PR.

API endpoints (tenant-scoped, admin-only except list)

GET    /v1/phone-numbers
GET    /v1/phone-numbers/available?country=&type=&contains=&limit=
GET    /v1/phone-numbers/pricing?country=
POST   /v1/phone-numbers/purchase     { phone_number, channel_id?, number_type?, monthly_price_usd? }
DELETE /v1/phone-numbers/:id
PATCH  /v1/phone-numbers/:id          { channel_id | null }

Legacy GET /v1/tenants/:tenant_id/phone-numbers is untouched — it serves the old shared-pool model, still used by some UI.

Webhook URLs

Purchases wire these URLs on the Twilio number:

voice_url        POST  https://api.twinlix.com/webhooks/twilio/voice?channel_id=<id>
statusCallback   POST  https://api.twinlix.com/webhooks/twilio/status?channel_id=<id>

channel_id in the query is what resolveWebhookAuthToken() uses to find the right subaccount token. Numbers without a channel_id (detached via PATCH) fall back to the master token.

Daily drift-sync cron

Registered in backend/src/index.ts as phone-numbers-drift-sync with schedule 15 3 * * * (03:15 server time). Two passes:

1. For every subscription with status='canceled' AND canceled_at <= NOW(), release all active numbers of that tenant.
2. For every tenant with an active subaccount, list their Twilio IncomingPhoneNumbers and mark locally-active rows without a match as released (drift). Numbers found only in Twilio are logged as orphans.

Grep phone_numbers_drift_sync in pm2 logs to see runs. Summary line includes tenantsChecked/tenantsFailed/driftReleased/bulkReleased/durationMs.

Cancellation flow

• subscriptionService.cancelSubscription(id, immediate=true) → sets canceled_at=NOW() and fires-and-forgets releaseAllTenantNumbers(tenant_id). HTTP response doesn't block on Twilio DELETEs.
• cancelSubscription(id, immediate=false) (end-of-period cancel) leaves the numbers in place until the cron picks them up after canceled_at passes.

Plan quotas

pricing_tiers.quota_phone_numbers bounds how many active numbers a tenant can hold. The purchase endpoint returns 402 with code=phone_number_quota_exceeded and used/quota when exceeded; the frontend surfaces the message as-is.

QA / E2E scenario (TWIN-40.18)

Run on prod or a staging aimed at the same Twilio master account. Buying one UK local number costs $1.15/month + per-minute, so clean up after.

1. Open https://app.twinlix.com/ → Setup → Integrations → open a voice channel's edit modal.
2. Confirm you see the new "Phone numbers" section with "No phone numbers yet" empty state.
3. Click + Connect voice line. In the modal:
   • Country: United Kingdom, Type: Local.
   • Click Search with no prefix.
   • Pick the first candidate, click Buy — $1.15/mo.
4. Toast: "Purchased +44..." — the list refreshes with the new row.
5. Back in the Twilio Console of the Twinlix master account, under Accounts → Subaccounts, the tenant's subaccount owns the new number; voice_url points at api.twinlix.com/webhooks/twilio/voice?channel_id=<id>.
6. Call the number from any mobile. Check pm2 logs:
   • Invalid Twilio signature should NOT appear — signature must verify with the subaccount token.
   • Inbound call received from twilio.ts:/voice.
   • 11Labs agent answers normally.
7. Hang up. Click Release on the row; confirm.
8. In our DB, phone_numbers row's status flips to released, released_at set. In Twilio Console the number is gone from the subaccount.

If step 6 fails with Invalid Twilio signature, cross-check: resolveWebhookAuthToken is fetching the right subaccount (channel_id must be present in the webhook URL).

Adding a new country to the UI

Edit twinlix-admin-front/src/components/1-ions/ConnectVoiceLineModal/index.tsx, COUNTRY_OPTIONS. Add { value: "CZ", label: "🇨🇿 Czechia" }. No backend changes needed — /available accepts any ISO-2 code Twilio supports.

Adding a new Twilio error code

Edit twinlix-admin-front/src/utils/twilioError.ts, CODE_MESSAGES map. Backend passes Twilio's code verbatim in response.data.twilio_code.

Troubleshooting

"Tenant already has a subaccount but the token doesn't decrypt"

The AES key changed or the ciphertext is corrupted. ensureSubaccount will throw rather than creating a parallel subaccount. Options:

• Restore the previous SUBACCOUNT_ENCRYPTION_KEY from backup.
• If truly unrecoverable: wipe the tenant's twilio_subaccount_auth_token_ciphertext + twilio_subaccount_sid in DB (write an SQL migration); next API hit provisions a new subaccount. The old one needs manual cleanup in Twilio Console.

"Twilio 400 on purchase: URL placeholders must match path_params_schema keys"

This is the 11Labs error wording but Twilio has similar validation for VoiceUrl. Check that the channel_id injected in the webhook URL is a valid UUID, not the literal {channel_id} placeholder.

Drift-sync reports orphans in Twilio

A number exists in the subaccount but not in our DB. Either a manual purchase in Twilio Console, or a DB-rollback that happened after the Twilio purchase succeeded. Investigate manually; the cron doesn't auto-adopt orphans.