Zoom webinar → registrants → ActiveCampaign + Close, in n8n. The registration path has been down since Sep 24. Zapier covered every registrant, so nothing was lost. Goal: full switch from Zapier to n8n this week.
Update Sep 29: issue 2 is resolved. 322a is adding new Zoom webinars again, confirmed with a real webinar. Webinars created before Sep 29 won't be backfilled.
| Workflow | State | Last 7 days |
|---|---|---|
322a New Zoom Webinar Created | active · fixed Sep 29 | Before the fix: 2 "ok" (both did nothing) · 3 error. Since the fix: first real webinar added Sep 29, 12:49 PM |
322d Upsert Webinar | active | 2,000+ ok · none since Sep 24 |
322b Webinar Scheduler | active | 0 runs |
323a Webinar Registration Hub | inactive · unpublished | 2,000+ ok · none since Sep 24 |
323b Register for Webinar in Zoom | active | 0 runs |
323c Add to Webinar Registration Log | active | 0 runs |
323e Zoom Registrant Webhook → AC Tag | active · shadow | 2,000+ ok |
323d Zoom Direct Registrant → AC Tag | active · draft | 143 ok · 2 error |
232b Lead Events in AC → Router (feeds 323a) | active | Successful runs aren't saved, so volume is unknown · 7 crashed Sep 25 · 1 error Sep 26 |
Kill switches in SETTINGS.webinar_modes: ZOOM_ADD, CRM_WRITES, BRAND_LOG, OUTBOUND_COMMS = true · REG_CUTOVER = false.
323a was saved without a published version on Sep 24 at 4:37 PM and hasn't run since. Its last run, 43 seconds earlier, was a real Gator registrant. The hub found no matching webinar in WEBINARS and logged it as "unmatched". Nobody on our side meant to turn it off. See the 323a plan.
232b crashed repeatedly on Sep 25 and hit a database connection timeout on Sep 26 (ledger (gator): "timeout exceeded when trying to connect"). The timeout is n8n running out of connections to its own database during a lead surge. 232b doesn't save successful runs, so we can't tell from n8n how many events it has handled since. See Fixing 232b under lead surges.
✓ Ruled out (Sep 28): the ActiveCampaign webhooks are confirmed active, so AC did not switch them off. With the webhooks live, 323a having no runs since Sep 24 points to 323a itself being unpublished.
Resolution (applied and published Sep 29):
Zoom now gets its reply first. The validation ping gets its token back in about 0.3 seconds. skip mint? was rewritten in a condition format the IF node reads. map_skip now carries the original Zoom data through the wait. Webinars Pulse already made end at a new execution_data_pulse_skip step that records the webinar and operator. Details are in Fixing 322a.
Verified: pinned tests passed for all three cases (Pulse-made, web portal, validation ping), with no writes. Then a real webinar created at 12:49 PM, "Business Buying with Pace Morby" (Oct 6), was added to WEBINARS with brand Subto, marked active, got its registrant sheet, and triggered the "needs SMS copy" nag.
Decision: webinars created before Sep 29 won't be backfilled.
What was wrong: a Pulse dedupe added on Sep 22 to skip webinars Pulse already created introduced three bugs:
The check always says "skip". skip mint? uses a condition format this version of the IF node doesn't read, so every webinar goes down the "skip" branch, which leads nowhere. Runs still show as "success".
Zoom gets no reply for 5 minutes. Zoom expects a reply within 3 seconds. Its URL-validation ping fails, which can make Zoom disable the webhook.
The original Zoom data is lost after the wait. Even if the check passed, the next step would read blank fields.
Missed while broken: "Deal Detective with Pace Morby" (Oct 1) and "Cash Flow with Co-Living" (recurring Oct 9–13). They won't be backfilled, per the Sep 29 decision.
322b, 323b and 323c have had 0 runs in 7 days. They depend on issues 1 and 2. Issue 2 is fixed, so they're now waiting only on issue 1 (323a).
Each step changes a production workflow, so each one needs Matt's approval.
ac_tag, and the hub matches registrants on that field, so decide who fills it in.✓ Applied and published Sep 29, and verified with a real webinar (see issue 2). The 5-minute wait and the Pulse dedupe stay; they now run after the reply to Zoom. No existing nodes were renamed.
If skip mint? says yes (Pulse already created the webinar), the run ends at a new execution_data_pulse_skip node that records the webinar name, id and operator email.
remove webhook → wait_5m
add webhook → zoom_validation
remove webhook_response → clean
add webhook_response → wait_5m
remove skip mint? [false] → zoom_validation
add skip mint? [false] → clean
add skip mint? [true] → execution_data_pulse_skip (new node)
const rows = $input.all().map(i => i.json).filter(r => r && r.status);
const hit = rows.length > 0;
return [{ json: {
...$('webhook').first().json, // body.payload.object.* for clean
skip_mint: hit,
map_status: hit ? rows[0].status : null,
webinar_id: $('map_check').first().json.webinar_id,
} }];
clean keeps reading $json.body.payload.object.* unchanged.
Replace the node in place (same name, same position) with IF typeVersion 2.2,
matching zoom_validation:
{ "conditions": {
"options": { "caseSensitive": true, "leftValue": "",
"typeValidation": "strict", "version": 2 },
"conditions": [{
"leftValue": "={{ $json.skip_mint }}", "rightValue": "",
"operator": { "type": "boolean", "operation": "true", "singleValue": true }
}],
"combinator": "and" },
"options": {} }
checked with n8n's validator: proposed = valid;
today = invalid ("conditions.conditions is missing"), so the node treats the check as passed:
{ "conditions": { "boolean": [{ "value1": "={{ $json.skip_mint }}", "operation": "isTrue" }] } }
a. validate_node_config on skip mint? and map_skip, then validate_workflow
b. test run with pinned data:
endpoint.url_validation → replies instantly with plainToken + encryptedToken
webinar.created (Pulse-made, id in pulse_event_map) → ends at pulse skip
webinar.created (web_portal, e.g. Deal Detective) → reaches clean with real fields
c. publish → Zoom Marketplace app → Event Subscriptions → re-validate the endpoint
d. backfill with 322c: Deal Detective (10/1) and Co-Living (10/9–13)
rollback: restore the version before the fix
Sep 29 results:
a ✓ valid
b ✓ all three cases routed correctly with every write pinned; no WEBINARS rows written
c ✓ published; live validation ping answered in 0.33 s; a real webinar
(Business Buying with Pace Morby, 12:49 PM) was added end to end.
Zoom is delivering events, so re-validating is optional.
d ✕ skipped: no backfill of webinars created before Sep 29
Not applied; needs Matt's approval.
Each write only runs if both the global switch in SETTINGS and that webinar's own flag in WEBINARS are true:
| Global (SETTINGS) | Webinar flag | Turns on |
|---|---|---|
| ZOOM_ADD ✓ | reg_platform | Register in Zoom and add the join link |
| CRM_WRITES ✓ | crm_writes | Close upsert, round robin, source, custom activity, Impact, AC email tag |
| BRAND_LOG ✓ | brand_log | Brand registration log and the "All Registrants" sheet |
| OUTBOUND_COMMS ✓ | confirm_sms | Confirmation text |
All four global switches are on, but every WEBINARS row I checked has these flags empty. So once published, the hub only claims, phone-verifies and logs. That's shadow mode, and it's the per-webinar switch for this week. REG_CUTOVER and the WEBINARS cutover column aren't read by the hub.
✕ Unpublished since Sep 24. Only one version exists in history, so there's nothing to compare against. A new workflow, 922h. NR Registry API, was created a minute earlier, which suggests a related change made outside this workspace.
✕ Registrants don't match a webinar. The hub looks webinars up by their AC tag. There's no WEBINARS row for "9/26/26 Gator Live Registrant", and 322a has dropped every new webinar since Sep 22. Those registrants go to unmatched_registrations and nothing else happens.
⚠ The SMS roster step isn't connected. append_registrant (the "SMS Verified" tab the SMS reminders read) has nothing feeding it, so the hub never adds anyone to the reminder list. The registrant-count update that follows it (zoom_count → update_reg_count) never runs either. The workflow's own note says OUTBOUND_COMMS should feed it.
⚠ No error alerts. 323a has no error workflow set, while 232b and 322a both use the shared error handler.
- What was the Sep 24 4:37 PM save? Is it tied to 922h NR Registry API?
- Was disconnecting append_registrant intentional (is Zapier or another flow
feeding the SMS Verified tab)?
- Should the switch stay per-webinar row flags, with REG_CUTOVER left unused?
settings.errorWorkflow = shared-error-handler
# only if Matt confirms the SMS roster should come from the hub:
add outbound_comms? [true] → append_registrant (alongside confirm_sms?)
remove append_registrant → append_all_registrants (already fed by
log_webinar_registration; avoids a double row)
keep append_registrant → zoom_count → count_ok? → update_reg_count
test_workflow with pinned "call" input (all row flags empty = shadow):
a. matched tag (e.g. "10/3/26 Gator Live Registrant") → claim → phone verify
→ stamp_completed; no Close/AC/Zoom calls
b. same person again → duplicate_stop
c. unknown tag → log_unmatched_registration
(the existing hub end-to-end test script covers these paths)
publish_workflow 323a
rollback: unpublish_workflow 323a
-- which AC tags arrived with no webinar row (marketing database)
SELECT tag, brand, count(*) AS registrants, min(created_at), max(created_at)
FROM unmatched_registrations
WHERE created_at > now() - interval '21 days'
GROUP BY tag, brand
ORDER BY registrants DESC;
Updated Sep 29: webinars created before Sep 29 (the Sep 26 Gator webinar and the 2 that 322a dropped) won't be backfilled. From Sep 29 on, 322a adds new webinars itself, but it leaves ac_tag empty. That field has to exactly match the AC tag before the hub can match registrants.
On that webinar's WEBINARS row, one flag at a time, checking runs after each:
1. reg_platform = true (Zoom add + join link)
2. crm_writes = true (Close, round robin, AC)
3. brand_log = true (brand log + All Registrants sheet)
4. confirm_sms = true (confirmation text)
then pause that webinar's Zapier zap.
Don't enable brand_log without crm_writes; log rows would have a blank lead/rep.
rollback: set the flags back to empty
The error comes from n8n's own internal database, not from a Postgres node in the workflow. Three things combine during a surge:
Tiny connection pool. DB_POSTGRESDB_POOL_SIZE is never set, so each n8n container gets the default of 2 connections, while each worker runs up to 10 executions at once.
One lead becomes many saved runs. 232b → ledger tap (923x) → 323a → about 11 sub-workflows. Each one writes a run record to the database.
Background load. 235w. Write Bus Drain runs every 2 seconds, about 43,000 runs a day, and each one uses the same database.
When connections run out, the webhook itself can fail. AC hasn't disabled the webhooks (confirmed active Sep 28), but repeated failures put that at risk. (n8n also returned a 502 "origin overloaded" once while this review was running.)
Nothing below has been applied. Each fix needs Matt's approval. Suggested order: A and B first (quick, lowest risk), then C, then D.
Biggest single improvement. The worst case is 11 containers × 10 = 110 connections. Confirm the database can take that before applying (see the pre-check).
# add to local.n8n_env (workers inherit it through worker_env)
# n8n's own DB pool. The default of 2 per process starved under AC lead
# surges on 9/25 ("timeout exceeded when trying to connect").
# 1 main + up to 10 workers x 10 = 110 connections max.
{ name = "DB_POSTGRESDB_POOL_SIZE", value = "10" },
{ name = "DB_POSTGRESDB_CONNECTION_TIMEOUT", value = "60000" },
# pre-check (n8n database): must comfortably exceed 110 + other clients
SHOW max_connections;
# CloudWatch: DatabaseConnections, CPUUtilization, CPUCreditBalance
# around 2026-09-25 11:56 AM Phoenix
# apply ONLY the n8n tasks/services, never the whole directory
terraform plan -target=aws_ecs_task_definition.main -target=aws_ecs_task_definition.worker \
-target=aws_ecs_service.main -target=aws_ecs_service.worker
terraform apply -target=... (same four targets)
# rollback: remove the two lines and re-apply the same four targets
If the database's CPU burst credits ran out on Sep 25, also move it from db.t4g.small to db.t4g.medium.
235w already keeps draining until the queue is empty within one run (up to 20 loops of 150 rows), so a 30-second schedule costs at most about 30 seconds of delay. It cuts about 43,000 runs a day to about 2,900.
node every_2s → rule.interval: [{ field: "seconds", secondsInterval: 30 }]
(keep the node name; only the interval changes)
rollback: set secondsInterval back to 2
Every lead starts an extra run just to log that it arrived. execution_data already records name, email, phone and ac_id on the same run. Disable the five ledger nodes, or replace them with a single direct Postgres insert into the ledger table. Either removes one saved run per lead.
disable: ledger (subto), ledger (zdb), ledger (gator), ledger (tc), ledger (sus)
(no renames; execution_data → universal_router path is untouched)
rollback: re-enable the five nodes
This is the fix for surges we can't control. 232b only saves the registrant to a queue table and returns straight away. A drainer pulls 20 every 30 seconds (about 2,400 an hour) and calls 323a one at a time. A surge makes the queue longer instead of causing failures, AC always gets a fast success reply, and every registrant has a row to audit.
It's a separate table because 235w claims every row in write_queue regardless of action, so it would pick up registrant rows and mark them unknown.
CREATE TABLE IF NOT EXISTS registrant_queue (LIKE write_queue INCLUDING ALL);
-- same columns, unique event_id, defaults. Note: shares write_queue's id sequence.
-- node: enqueue_registrant [postgres · executeQuery · marketing database]
-- wired: Registrant tag? (true) → enqueue_registrant
-- keep the old Execute node, disabled, for rollback
WITH t AS (SELECT $1::jsonb AS j)
INSERT INTO registrant_queue (event_id, platform, account, action, priority,
coalesce_key, request_cost, payload, status, attempts, last_error)
SELECT
'reg|' || (j->>'brand') || '|' || (j->>'contact_id') || '|' || (j->>'tag'),
'n8n', j->>'brand', 'webinar_register', 'realtime',
'reg|' || (j->>'brand'), 1, j, 'queued', 0, ''
FROM t
WHERE coalesce(j->>'contact_id', '') <> ''
ON CONFLICT (event_id) DO NOTHING
RETURNING event_id;
-- queryReplacement (same 14 fields 323a gets today):
={{ JSON.stringify({ ac_id: $json.contact_id, contact_id: $json.contact_id,
email: $json.email, phone: $json.phone, fname: $json.first_name,
lname: $json.last_name, tag: $json.tag, brand: $json.brand,
utm_source: $json.utm_source, utm_medium: $json.utm_medium,
utm_campaign: $json.utm_campaign, utm_content: $json.utm_content,
utm_term: $json.utm_term, impact_click_id: $json.impact_click_id,
impact_partner_id: $json.impact_partner_id }) }}
-- settings: retryOnFail true, maxTries 3, waitBetweenTries 2000
The event id is built from brand, contact and tag, so if AC resends the same registration it's ignored (today's random id can't catch that).
every_30s [schedule]
→ claim [postgres]
→ Loop Over Items (batch 1)
→ register [executeWorkflow 323a, waitForSubWorkflow: true,
onError: continueErrorOutput]
ok → mark_done → execution_data (name, phone, email)
err → mark_retry → execution_data (name, phone, email)
settings: executionTimeout 600, errorWorkflow shared-error-handler,
saveDataSuccessExecution none
-- claim: 20 rows, skips if a batch is still in flight
WITH inflight AS (
SELECT count(*) AS n FROM registrant_queue
WHERE status = 'working' AND updated_at > now() - interval '2 minutes'
), cand AS (
SELECT id FROM registrant_queue
WHERE attempts < 5
AND (status = 'queued'
OR (status = 'working' AND updated_at < now() - interval '10 minutes'))
AND (not_before IS NULL OR not_before <= now())
AND (SELECT n FROM inflight) = 0
ORDER BY id
LIMIT 20
FOR UPDATE SKIP LOCKED
)
UPDATE registrant_queue q SET status = 'working', updated_at = now()
FROM cand WHERE q.id = cand.id
RETURNING q.id, q.event_id, q.payload, q.attempts;
-- mark_done
UPDATE registrant_queue
SET status = 'done', finished_at = now(), updated_at = now()
WHERE id = $1;
-- mark_retry: exponential backoff, gives up after 5 attempts
UPDATE registrant_queue
SET attempts = attempts + 1,
status = CASE WHEN attempts + 1 >= 5 THEN 'failed' ELSE 'queued' END,
last_error = left($2, 500),
not_before = now() + (power(2, least(attempts + 1, 5)) || ' seconds')::interval,
updated_at = now()
WHERE id = $1;
Tuning: raise LIMIT or shorten the interval for faster throughput. A stuck row is retried after 10 minutes. Rollback: re-enable the old Register node in 232b and disable enqueue_registrant. Any queued rows can be drained later.
| Item | Issue |
|---|---|
333a WJ Registrant Poller | Every write fails because the "Append All Registrants" column setup is out of date. Missing columns: registered_at, fname, lname, join_url, lead_id, rep, sms_verified. |
235d Lead → Add Tag in AC | Intermittent 500 errors from the Gator AC account (Signed Up → student tag). |
| Migration status page | Lists 343a AMA as "live poll", but that workflow is inactive. |