The best design is a read-only, per-account Gmail sync worker that treats Gmail historyId as the primary incremental cursor, keeps a bounded local durable state file as the source of truth for each account, and stores only message metadata, snippets, selected headers, classifications, and hashes unless a particular message must be summarized. The daily digest should be generated separately for briguy100 and brian.c.keefe, then merged only at the presentation layer.
For Brian’s current tokens, both accounts appear to have https://www.googleapis.com/auth/gmail.readonly plus Calendar scope. That is enough to list messages, read headers/snippets/bodies when needed, call Gmail history APIs, classify new mail, build digests, learn ignore preferences locally, and surface unsubscribe candidates. It is not enough to mark mail read, apply labels, archive, unsubscribe, send replies, create drafts, or modify Gmail settings.
Sourced facts: Gmail exposes message IDs, thread IDs, labels, snippets, historyId, and internalDate; Gmail history records are chronological but not contiguous; old or invalid startHistoryId values typically return HTTP 404 and require full sync; message listing supports Gmail search syntax but not under gmail.metadata; message retrieval supports format=METADATA plus selected headers; gmail.readonly permits viewing email messages/settings, while gmail.modify and gmail.send are broader; RFC 2369 defines list command headers such as List-Unsubscribe; RFC 8058 defines one-click unsubscribe using List-Unsubscribe-Post: List-Unsubscribe=One-Click; Google’s sender guidance says bulk marketing/subscribed messages must support one-click unsubscribe and include a visible unsubscribe link.
Design recommendations: Keep separate account namespaces; use historyId as primary cursor plus a recent internalDate overlap scan as recovery insurance; use local state and a processed-message ledger for idempotency; never click unsubscribe links automatically; minimize body retention; and keep user-taught ignore rules explainable, reversible, and easy to audit.
Brian wants AI agents to use read-only access to two Gmail accounts:
briguy100brian.c.keefeRecommended architecture:
1. Account registry: two configured accounts, each with a token file at /opt/data/google-accounts/{account}/google_token.json. The implementation should reference these paths but never log token contents.
2. Per-account sync worker: one logical worker loop per account, same code path, separate state namespace.
3. Local durable store: SQLite is preferred for ledgers/rules/candidates; JSON is acceptable for the small cursor file. Store under a private Hermes data directory with restrictive permissions.
4. Classifier/ranker: deterministic heuristics first, LLM classification second only for candidate messages that survive ignore/noise filters.
5. Digest renderer: daily Telegram message and optional host-served page. The Telegram digest should clearly separate accounts and keep a short format.
6. Weekly hygiene report: subscription/noise candidates only; no modifications to Gmail.
Privacy posture: the mailbox remains the source of truth; the agent stores durable metadata and learned preferences, not a replica of email bodies.
historyId, internalDate/message ID, labels, and local statehistoryIdGmail’s own sync guide recommends full sync first, then partial sync with users.history.list using a stored startHistoryId. History returns changes newer than that value, including added messages and label changes. The method’s response includes the mailbox’s current historyId; if there is no nextPageToken, the client can store that returned value for a future request. Gmail also states that history IDs increase chronologically but are not contiguous and can have random gaps.
Use it this way:
last_history_id from a recent message or the profile/history response.users.history.list(startHistoryId=last_history_id, historyTypes=messageAdded,labelAdded,labelRemoved).format=METADATA headers first; fetch full body only if ranking/summarization requires it.last_history_id only after all pages are processed and local state is committed.Why it is best: it catches message additions and label changes without relying on timestamps, and it is the native Gmail sync mechanism.
internalDate plus message ID overlapGmail’s Message resource includes immutable id, threadId, labels, snippet, historyId, and internalDate. internalDate is useful for backfills and overlap scans, but it is weaker than history as the main cursor because:
Recommendation: keep last_seen_internal_date_ms and always do a small overlap scan, for example newer_than:3d or after:YYYY/MM/DD with local de-dupe by message ID. Treat it as an insurance layer, not the primary source of truth.
message.id is the immutable per-message identity and should be the main processed ledger key.threadId groups related messages for digest display and dedupe of repeated thread updates.Labels are useful features, not sufficient cursors. With read-only access the agent can observe labels such as INBOX, UNREAD, CATEGORY_PROMOTIONS, CATEGORY_UPDATES, SPAM, TRASH, or user labels, but cannot apply or remove labels. Use labels for classification and filtering; do not rely on labels alone to know what was already processed.
Local state is mandatory because the agent needs cross-run memory: cursor values, resync status, processed IDs, learned rules, and digest history. The Gmail API tells what changed; the local state tells what the agent has already acted on.
Gmail says history records are typically available for at least one week and often longer, but the period can be shorter; invalid or out-of-date startHistoryId usually returns HTTP 404, and the client must perform a full sync.
Recommended behavior:
1. Normal run: process history pages from last_history_id to returned current history ID.
2. Missed run but history still valid: process the larger batch normally; cap digest length and move overflow into “more items” summary.
3. HTTP 404/out-of-date history: enter resync_required; perform bounded full sync over a safe window, e.g. 14 or 30 days for digest use, plus all unread inbox messages if desired.
4. Ledger-preserving resync: never clear the processed ledger. Re-read recent messages and skip already processed message IDs.
5. Advance cursor after resync: store a fresh current history ID after the bounded full sync succeeds.
6. Alert Brian: Telegram note: “Gmail digest recovered from expired cursor; scanned last 30 days; no mail modified.”
A truly complete mailbox sync is possible, but for Brian’s use case it is unnecessary and privacy-worse. Prefer bounded resync plus explicit warning if a gap may remain.
Use three layers:
1. Primary: (account_id, message_id) unique key.
2. Thread display: (account_id, thread_id, digest_date) to collapse multiple new messages into one digest item.
3. Cross-account duplicate detection: hash normalized (Message-ID header, From, Subject, internalDate bucket) to flag newsletters or alerts that hit both accounts; do not drop cross-account items silently because the account destination matters.
Late-arriving or imported mail should be caught by the overlap scan. Each run should scan a short recent window in addition to history and reconcile any unprocessed message IDs.
Store new-message records separately from thread summaries. For a digest, roll up thread-level items:
Initial backfill should be explicit and bounded:
Backfills should mark messages as processed without sending an enormous digest unless emit_backfill_digest=true.
If daily digest fails for several days, next run should:
Use America/New_York for digest windows and display. Store UTC timestamps internally. Cursor logic should not depend on local dates except for presentation and bounded Gmail query construction.
Every run should have a run_id. A digest item is emitted once per (account_id, message_id, digest_kind) unless a later thread update changes the thread. Writes to state should be transactional: ingest → classify → commit ledger → render digest → record delivery outcome. If Telegram delivery fails after state commit, retain a pending_delivery row and retry the same digest without reclassifying.
Brian should be able to teach the system rules like:
Rule dimensions:
List-IDFalse-positive safeguards:
1. Never auto-ignore high-risk classes: security alerts, password resets, MFA, account changes, invoices above threshold, legal/tax/medical/financial, deadlines, travel changes, human replies.
2. Quarantine mode for new rules: first 3–5 matches go into “would ignore” digest section before silent suppression.
3. Explainability: each hidden item stores matched_rule_id and reason.
4. Feedback loop: Telegram actions can be represented as commands: “show more from X,” “always ignore this sender,” “undo rule,” “promote this category.” With current read-only Gmail scopes, feedback updates only local rules, not Gmail.
5. Decay/review: rules with no matches for 90–180 days can be archived after review; noisy rules with many exceptions should be re-suggested for refinement.
The agent should produce a periodic unsubscribe-candidate list, not unsubscribe or modify mail. Inputs:
List-Unsubscribe and/or visible unsubscribe text;Candidate output fields:
safe_to_review, suspicious, no_header, transactional_do_not_unsubscribeDo not include raw one-click URLs in Telegram if they are long/tokenized; link them from a secure local/private page or present the sender and a “review in Gmail” instruction. Tokenized unsubscribe URLs can reveal recipient identity.
RFC 2369 defines list command headers, including List-Unsubscribe, so mail clients can provide common UI for mailing-list functions. RFC 8058 was created because automated software sometimes fetches unsubscribe URLs in headers and accidentally triggers unsubscriptions. RFC 8058 requires senders that support one-click to include both a List-Unsubscribe header with an HTTPS URI and List-Unsubscribe-Post: List-Unsubscribe=One-Click, and the one-click POST encodes recipient/list identity in the URI.
Google’s bulk sender guidance says marketing and subscribed messages must support one-click unsubscribe and include a visible unsubscribe link; senders over the bulk threshold must include both List-Unsubscribe-Post: List-Unsubscribe=One-Click and List-Unsubscribe: <https://...> headers. The same guidance ties deliverability to authentication and anti-spoofing practices such as SPF/DKIM/DMARC alignment.
Design implication: an AI agent should surface unsubscribe candidates, not blindly click links.
Reasons:
Best UX: “Candidate: Example Deals — 38 messages/90d — List-Unsubscribe present — review in Gmail / copy safe link / ignore sender.” Require Brian confirmation before any future modify/send-scope unsubscribe automation.
Recommended priority order:
1. Action-required: direct questions, requests, forms to complete, approvals, scheduling asks, inbound opportunities, messages with explicit asks.
2. Waiting-on-me: replies in threads where Brian was the last expected actor, stale threads with open requests, messages containing “following up.”
3. Deadlines/time-sensitive: due dates, expiring links, travel changes, calendar invites in email, event reminders, bills due.
4. Receipts/security/account alerts: receipts, invoices, login alerts, password/MFA/account changes, financial alerts. Security should always surface; routine receipts can be summarized.
5. Human FYI: personal/professional updates with no action.
6. Newsletters: subscribed content Brian may want but usually not urgent.
7. Marketing/promotions: sales outreach, offers, product announcements.
8. Noise: low-value recurring notifications, duplicates, social updates, spam-like content.
Classifier inputs should include headers, sender, recipients, subject, snippet, labels, List-ID, unsubscribe headers, internal date, and minimal body excerpt only when needed. For professional mail, raise weights for client/customer/vendor/recruiter domains; for personal mail, raise security, finances, family/friends, travel, healthcare, and household operations.
Recommended retention policy:
Fetch strategy:
1. messages.list to get IDs.
2. messages.get(format=METADATA, metadataHeaders=[From, To, Cc, Date, Subject, Message-ID, In-Reply-To, References, List-ID, List-Unsubscribe, List-Unsubscribe-Post, Reply-To, Authentication-Results]).
3. Use snippet if available.
4. Fetch format=FULL only for top-ranked ambiguous messages that require actual summarization.
5. Store body-derived summaries, not bodies, unless Brian explicitly opts in.
`json
{
"account_id": "briguy100",
"email_hint": "briguy100",
"token_path": "/opt/data/google-accounts/briguy100/google_token.json",
"scope_fingerprint": "sha256-of-sorted-scopes-not-token",
"last_history_id": "1234567890",
"last_history_advanced_at": "2026-07-20T12:05:00Z",
"last_seen_internal_date_ms": 1784568300000,
"last_successful_run_id": "2026-07-20T12:00:00Z_briguy100",
"resync_status": "ok",
"resync_reason": null,
"overlap_days": 3,
"backfill_window_days": 30,
"timezone": "America/New_York"
}
`
`json
{
"rule_id": "rule_20260720_001",
"scope": {"accounts": ["briguy100"], "applies_to": "message"},
"match": {
"from_email": null,
"from_domain": "example.com",
"list_id": null,
"subject_regex": "(?i)weekly deals",
"gmail_labels_any": ["CATEGORY_PROMOTIONS"]
},
"action": "suppress_from_daily_digest",
"exceptions": {
"keywords_any": ["security", "password", "invoice", "due", "urgent"],
"amount_over_usd": 100
},
"confidence": 0.82,
"status": "quarantine",
"created_from_feedback": "Brian: ignore these promos",
"created_at": "2026-07-20T12:00:00Z",
"last_matched_at": null,
"review_after_matches": 5,
"false_positive_count": 0
}
`
`json
{
"account_id": "brian.c.keefe",
"message_id": "18fabc123",
"thread_id": "18fabc999",
"rfc822_message_id_hash": "sha256:...",
"internal_date_ms": 1784568300000,
"history_id_at_ingest": "1234567890",
"from_domain": "company.com",
"subject_hash": "sha256:...",
"labels_seen": ["INBOX", "UNREAD", "CATEGORY_PRIMARY"],
"classified_as": "action_required",
"rank_score": 0.91,
"digest_emitted_at": "2026-07-20T12:10:00Z",
"matched_rule_ids": [],
"body_retention": "none",
"created_at": "2026-07-20T12:05:00Z"
}
`
`json
{
"dedupe_key": "sha256:normalized-message-id-or-sender-subject-date",
"account_ids": ["briguy100", "brian.c.keefe"],
"message_ids": ["msg_a", "msg_b"],
"thread_ids": ["thr_a", "thr_b"],
"first_seen_at": "2026-07-20T12:05:00Z",
"decision": "show_per_account_but_mark_duplicate"
}
`
`json
{
"candidate_id": "unsub_20260720_001",
"account_id": "briguy100",
"sender": "Deals <[email protected]>",
"from_domain": "example.com",
"list_id": "deals.example.com",
"volume_30d": 18,
"volume_90d": 61,
"last_received_at": "2026-07-19T18:22:00Z",
"sample_subjects": ["Weekend deals", "Last chance"],
"list_unsubscribe_present": true,
"list_unsubscribe_post_one_click": true,
"visible_unsubscribe_detected": true,
"risk_flags": ["tokenized_url_redacted"],
"recommendation": "review_unsubscribe",
"status": "candidate",
"brian_feedback": null
}
`
Keep it concise and separated by account:
`text
Gmail digest — Mon 8:00 AM ET
Personal / briguy100
! 2 action · 1 security · 3 receipts · 7 newsletters hidden
1. ! Dentist form due Wed — complete intake link
2. 🔐 Google security alert — new sign-in check
3. $ Amex receipt summary — 3 receipts, $84 total
Professional / brian.c.keefe
! 3 action · 2 waiting · 1 deadline · 12 low-priority hidden
1. ! Acme: contract redlines — asks for response today
2. ↩ Waiting on me: Sam follow-up from Fri
3. ⏰ Vendor renewal — due Jul 25
Noise: 19 suppressed by learned rules. Weekly unsubscribe review: 6 candidates.
Reply: more personal / more work / teach ignore <#> / undo rule
`
For long digests, send a short Telegram summary with a private page link. Never paste sensitive bodies into Telegram by default.
Weekly report sections:
Observed token metadata for both configured accounts indicates:
https://www.googleapis.com/auth/gmail.readonlyhttps://www.googleapis.com/auth/calendarWith gmail.readonly, the agent can:
List-Unsubscribe headers and classify mail;It cannot:
Future scopes:
gmail.metadata: more privacy-minimal but cannot use q search and cannot read bodies; useful only if the MVP is strictly header-based.gmail.modify: needed for labels, read/archive/trash, and many mailbox-management actions. Google describes it as read, compose, and send access but not immediate permanent deletion.gmail.send: send mail on Brian’s behalf; only needed for replies/outbound notifications from Gmail.gmail.compose: create/manage drafts and send; useful if Brian wants draft replies, but it is broader than read-only.mail.google.com: full Gmail control including permanent deletion; should be avoided unless absolutely required.Recommendation: keep read-only for the MVP and v1. If Brian later wants Gmail labels like AI Digest/Action Required or automatic archiving, request gmail.modify only after a separate approval and dry-run period.
Use the existing token locations:
/opt/data/google-accounts/briguy100/google_token.json/opt/data/google-accounts/brian.c.keefe/google_token.jsonDo not print, copy, or store token contents. At startup, read only token metadata needed to initialize OAuth credentials and compute a non-secret scope fingerprint.
Recommended files:
`text
/opt/data/gmail-digest-agent/
config.yaml
state/accounts/briguy100.cursor.json
state/accounts/brian.c.keefe.cursor.json
state/gmail_digest.sqlite
logs/daily.log
exports/latest-digest.md
`
Implementation steps:
1. Build a small Python Gmail client around google-api-python-client or the existing Hermes Google Workspace helper, parameterized by token path.
2. Add AccountConfig for each account: account ID, token path, display label, timezone, digest preferences.
3. Bootstrap each account with bounded backfill: last 7–30 days plus unread inbox.
4. Store cursor JSON and SQLite ledger transactionally.
5. Implement history sync; on 404, run bounded resync.
6. Fetch metadata headers first; fetch body only for high-value ambiguous messages.
7. Apply learned rules and deterministic ranking.
8. Generate daily Telegram text and optional page.
9. Generate weekly unsubscribe candidate review.
10. Add observability: run summary, counts, API errors, cursor age, resync events, and token/scope status without secrets.
Track per account:
Failure policy:
List-ID, List-Unsubscribe, List-Unsubscribe-Post.Implement the MVP with read-only Gmail access, per-account historyId cursors, overlap recovery scans, a local SQLite ledger, and explainable local rules. Keep the accounts separate in storage and presentation. Use the daily digest for high-signal awareness and the weekly report for subscription hygiene. Do not click unsubscribe links or mutate Gmail until Brian explicitly opts into broader scopes and a dry-run period proves the classifier is safe.