МАТЧАСТЬ / TECHNICAL ARCHITECTURE / DOCUMENT 40 / 03.09.2026

Техническая архитектура «Матчасти»

Рекомендуемая production-архитектура для первого реального масштаба продукта: публичное медиа и реестр, CMS/moderation, компании и эксперты, Agency Workspace, Search Proof, AI Visibility, Before/After, Reputation Layer, Next Best Action и billing. Основное решение: не строить микросервисный зоопарк. Запускать модульный монолит с отдельными асинхронными workers на одном хорошо изолированном Docker-host, используя PostgreSQL как главный источник истины, Redis как очередь/временное состояние, S3-compatible object storage для файлов и Authentik для identity provider.

Modular Monolith13 доменных модулей, одна транзакционная система, чёткие внутренние границы

Async by defaultcrawling, AI runs, Search APIs, reports и media processing вне HTTP request path

Postgres firstSQL + FTS + pg_trgm + pgvector до появления измеренной причины добавлять отдельные базы

Single-host firstDocker Compose на текущем сервере; Kubernetes, Kafka, Neo4j и ClickHouse пока не нужны

1. Главное архитектурное решение

Модульный монолит + отдельные worker processes. HTTP/API и транзакционная бизнес-логика остаются в одном backend-коде и одной основной PostgreSQL-базе. Долгие или внешние операции уходят в очереди. Это сохраняет транзакционную целостность, ускоряет разработку и оставляет понятный путь к будущему выделению сервисов.

INTERNET │ ▼ nginx / TLS / routing │ ├──────────────► Authentik │ ├──────────────► Next.js Web │ │ │ ▼ └──────────────► Fastify API │ ┌───────────┼────────────┐ ▼ ▼ ▼ PostgreSQL Redis MinIO/S3 │ │ │ └──► BullMQ Workers │ ├─ crawler │ ├─ search-proof │ ├─ AI visibility │ ├─ reports │ ├─ media │ └─ notifications │ └── transactional outbox → async jobs/events

2. Почему не микросервисы

На старте микросервисы добавят: distributed transactions service discovery more deployments cross-service auth versioned contracts more observability more failure modes more DevOps work WITHOUT: proven scaling need.

Нельзя путать широкий функционал продукта с необходимостью иметь 20 сервисов.

3. Когда модульный монолит перестаёт быть достаточным

Extract a service only if: • independent scaling is repeatedly needed • failure isolation materially matters • deployment cadence diverges • database/write pattern conflicts • a team can own the boundary • measurements prove bottleneck Not because: "microservices are modern".

4. Текущая инфраструктурная база

Single server: Ubuntu 24.04 LTS Intel i7-13700F ≈64 GB RAM RTX 4080-class 16 GB VRAM NVMe ≈2 TB Docker nginx Already available in environment: PostgreSQL / pgvector Redis MinIO Authentik.

5. Решение по изоляции

Mathchast получает отдельный production stack и отдельные state boundaries.

Shared host infrastructure: nginx Authentik host monitoring/backups Mathchast dedicated: app network web api workers PostgreSQL instance/volume Redis instance database credentials MinIO bucket/service account logs/telemetry namespace.

6. Почему dedicated Postgres/Redis, а не общий namespace

На одном физическом сервере дополнительный container overhead намного меньше operational blast radius от чужого проекта. Отдельный Postgres volume/user/config и отдельный Redis позволяют независимо делать upgrades, resource limits, backups и incident recovery.

7. MinIO можно разделить логически

Shared MinIO deployment допустим, если: bucket: mathchast-prod service account: mathchast-app policies: bucket-scoped versioning/lifecycle: Mathchast-specific backup: independently restorable.

8. Authentik — shared infrastructure

Authentik официально поддерживает OAuth2/OIDC, SAML и proxy providers. Для собственного приложения «Матчасти» рекомендуемый protocol — OIDC authorization code flow; application authorization всё равно хранится в Mathchast domain model.

9. Почему Authentik не должен хранить все роли продукта

Authentik: WHO is user? Mathchast DB: WHAT can user do inside: company workspace agency publication billing verification.

10. Monorepo

mathchast/ ├─ apps/ │ ├─ web/ │ ├─ api/ │ ├─ worker/ │ ├─ scheduler/ │ └─ admin/ optional route/app ├─ packages/ │ ├─ domain/ │ ├─ db/ │ ├─ contracts/ │ ├─ auth/ │ ├─ telemetry/ │ ├─ ui/ │ ├─ config/ │ └─ testing/ ├─ infra/ │ ├─ docker/ │ ├─ nginx/ │ ├─ migrations/ │ └─ scripts/ └─ docs/adr/

11. Основной runtime language

TypeScript end-to-end для web/API/core workers. Python добавляется только там, где AI/data ecosystem действительно даёт преимущество.

12. Почему не Python backend целиком

Python отлично подходит для model/data workers, но единый TypeScript stack уменьшает количество runtime boundaries для web product. Это project decision, а не утверждение о превосходстве языка.

13. Web

Next.js Responsibilities: public pages SSR/SEO workspace UI forms same-origin browser experience route-level caching public metadata structured data rendering.

Current Next.js documentation describes it as a React framework for full-stack web applications. Для «Матчасти» мы используем его прежде всего как presentation/SSR layer, не как контейнер всей domain logic.

14. Core API

Fastify + TypeScript Responsibilities: domain commands queries authorization transactions billing state publication state workspace state public/private API contracts.

15. Почему отдельный API, если Next.js умеет full-stack

Need: workers reuse domain services stable internal API boundary future agency API central authz long-lived product logic independent API tests Therefore: Next = UI/BFF-ish layer API = domain boundary.

16. API не обязан быть публичным

MVP: browser → same domain /api/* nginx → internal API P2: api.mathchast.com partner API versioned external contracts.

17. API style

REST JSON + OpenAPI. GraphQL не нужен на MVP.

Conventions: resource-oriented URLs explicit commands when state machine requires cursor pagination idempotency keys for writes request_id problem-style errors OpenAPI contract typed generated clients optional.

18. Почему не GraphQL

No demonstrated need for: arbitrary client graph queries many external consumers REST is simpler for: authz caching observability webhooks agency API later.

19. PostgreSQL — source of truth

По состоянию на август 2026 текущая поддерживаемая ветка PostgreSQL — 18; официальная документация показывает 18.6. PostgreSQL уже имеет full-text search, JSONB, row-level security, partitioning и большое количество зрелых indexing capabilities.

20. Версия PostgreSQL

Не требуется мигрировать production на конкретную версию только ради документа. Перед launch выбрать текущую поддерживаемую stable version, проверить pgvector compatibility и зафиксировать image tag.

21. Public IDs

Preferred: UUIDv7 If PostgreSQL 18: native uuidv7() available Alternative: application-generated UUIDv7 Benefits: non-guessable timestamp-ordered.

22. Database roles

mathchast_owner migrations only mathchast_app normal API mathchast_worker scoped worker access mathchast_readonly analytics/support optional backup role backup only.

23. App role is NOT table owner

PostgreSQL documentation notes that table owners normally bypass Row-Level Security and roles with BYPASSRLS always do. Поэтому migration owner и runtime application role должны быть разными.

24. Database schemas

identity verification publishing editorial distribution searchproof analytics reputation visibility recommendations commerce agency research platform.

25. Schema boundaries

Они нужны прежде всего для ownership/clarity, не как псевдомикросервисы. Cross-schema foreign keys допустимы в рамках осознанных domain relations.

26. Core module map

Module	Owns
Identity	companies, brands, experts, products, aliases, relations
Verification	claims, sources, evidence, verification sessions
Publishing	drafts, versions, publications, slugs, redirects
Editorial	moderation, review states, policies, assignments
Distribution	topics, recommendations surfaces, newsletters/events
Search Proof	health snapshots, crawlers, GSC/Yandex observations
Visibility	prompt sets, runs, answers, mentions, citations
Reputation	reviews, references, credentials, media portfolio
Recommendations	gaps, actions, evidence, outcomes
Commerce	orders, credits, invoices, refunds, metering
Agency	organizations, workspaces, delegation, client isolation
Research	studies, datasets, snapshots

27. JSONB policy

Use relational columns for: IDs state money permissions relations dates metrics used in queries Use JSONB for: provider raw payload variable metadata render config rare extension fields.

28. Почему не «всё JSONB»

PostgreSQL JSONB хорошо индексируется и обрабатывается, но entity relations, billing, permissions и measurements требуют строгих constraints and predictable queries.

29. Immutable raw observations

AI provider responses, Search API snapshots and external observations store raw payload separately from parsed metrics.

raw observation immutable parser v1 → mentions/citations parser v2 later → recompute Original evidence: preserved.

30. Derived metrics are recomputable

Store: raw answer raw citation URLs run metadata Derived: Mention Rate SoV Citation Rate Narratives Metric version: explicit.

31. High-volume tables

Candidates: ai_runs ai_answers crawler_observations search_daily analytics events audit logs Start: normal tables + indexes Partition: only after size/query evidence.

32. PostgreSQL partitioning policy

PostgreSQL itself recommends careful use; partitioning pays off mainly for sufficiently large tables and suitable access patterns. Поэтому не partition every table on day one.

33. Likely first partitions

If needed: ai_observations by month crawler_observations by month first-party raw events by month audit log by month/year Keep: entities/publications/orders unpartitioned.

34. Search stack: PostgreSQL first

MVP не требует Elasticsearch/OpenSearch/Meilisearch.

Exact: B-tree Full text: tsvector + GIN Fuzzy names: pg_trgm Semantic candidate retrieval: pgvector Final ranking: business rules + text relevance.

35. PostgreSQL FTS

PostgreSQL 18 has a complete full-text-search subsystem: document parsing, tsquery/tsvector, ranking, highlighting, dictionaries and GIN/GiST indexing.

36. pg_trgm

pg_trgm is an official supplied extension for trigram similarity and index-supported fuzzy matching. Это подходит для названий компаний, алиасов и опечаток.

37. pgvector

pgvector provides vector similarity search directly in PostgreSQL and supports standard distance operators and approximate indexes. Current repository documentation shows v0.8.x and packages for supported PostgreSQL versions.

38. Public search ranking v1

1. exact verified company name 2. exact alias/brand 3. prefix 4. trigram similarity 5. full-text topic/material 6. semantic fallback Boost: verified/current/public not: paid plan.

39. Search payment neutrality

Оплата не должна автоматически повышать entity/company search ranking. Это отдельный trust and editorial rule.

40. Когда добавить Meilisearch/OpenSearch

Only if measurements show: public search latency poor relevance tuning blocked facets become complex index load hurts Postgres autocomplete load too high Then: external search is read model, Postgres remains source of truth.

41. No Neo4j on MVP

Entity Graph in Docs 19–20 does not require a graph database. Typed relational edges + recursive SQL + pgvector candidate retrieval are sufficient initially.

42. Content source format

Structured block document as source of truth, sanitized HTML as rendered derivative.

publication_version: content_json plain_text rendered_html render_version word_count source_refs entity_refs created_at.

43. Editor technology

ProseMirror/Tiptap-class editor or equivalent Need blocks: paragraph heading list quote image table callout source verified claim embed disclosure.

44. Не хранить только HTML textarea

Structured content simplifies source blocks, validation, machine readability, redesign, exports and future editor migrations.

45. Published versions immutable

Draft edits: mutable draft version Publish: freeze version N Correction: version N+1 Public URL: same publication_id history retained.

46. Stable URL model

publication_id: immutable slug: human-readable slug change: redirect table canonical: current public URL history: preserved.

47. Public page delivery

Request: nginx → Next.js Next: read public view/API render: HTML + metadata + schema Cache: public only short/revalidated Private: no shared cache.

48. Public pages must render without client JS dependency

Article/company/research content, canonical, metadata and structured data are server-rendered. JavaScript enhances interaction, not discoverability.

49. Cache invalidation event

PublicationPublished PublicationCorrected CompanyUpdated → invalidate page cache → update sitemap state → enqueue IndexNow → refresh search document → distribution event.

50. Event architecture

PostgreSQL transactional outbox + BullMQ.

DB transaction: update publication insert outbox_event COMMIT Dispatcher: reads outbox enqueues BullMQ marks dispatched Workers: idempotent.

51. Почему outbox

Without: DB commit succeeds queue publish fails → inconsistent state With outbox: domain event belongs to same transaction.

52. Redis/BullMQ role

BullMQ is a Redis-backed queue library designed for workers, delayed jobs, retries, concurrency and crash recovery. Its documentation notes at-least-once delivery can occur in worst-case conditions.

Следствие: worker handlers обязаны быть idempotent.

53. Job identity

job_key examples: publication:123:indexnow:v7 ai-run:456:chatgpt:replicate:2 health:publication:123:2026-09-03 report:789:v1 Duplicate: safe no-op / upsert.

54. Queue classes

critical default crawl search-sync ai gpu media report email low-priority.

55. Не одна очередь для всего

AI provider timeout must not block password email or publication state job.

56. Worker processes

worker-core: small jobs worker-crawl: HTTP external worker-search: GSC/Yandex/IndexNow worker-ai: provider adapters/parsing worker-gpu: local models worker-report: PDF/report generation worker-media: images/files.

57. Scheduler

Schedule definitions live in PostgreSQL; scheduler periodically enqueues due work. Queue system executes work, but DB remains audit source for why/when a check was scheduled.

58. Why not hidden cron only

Need: client-visible next run plan quotas pause replay audit provider rate limits window semantics.

59. Schedule row

schedule_id type subject_id next_run_at frequency priority quota_scope enabled locked_until last_run_id.

60. Scheduler concurrency

SELECT due rows FOR UPDATE SKIP LOCKED LIMIT N enqueue advance next_run_at Supports: multiple scheduler instances later.

61. External provider adapter pattern

AIProvider SearchProvider PaymentProvider EmailProvider StorageProvider Each: normalized interface raw response storage rate limit retry policy health.

62. AI provider record

provider platform model/mode if known region language collection_method request_version response_id started_at completed_at cost raw_object_key status.

63. Do not hard-code «ChatGPT response» schema

Providers differ and change. Normalized metrics live above adapters; raw provider payload remains provider-specific.

64. AI collection pipeline

Measurement plan → scheduled run → provider request → raw response → parse answer/citations → entity resolution → metric observations → quality checks → snapshot/report.

65. Local GPU use

RTX 4080 can be useful, but local AI is an optimization/worker, not a hard dependency of public product.

Good local tasks: embeddings classification entity candidate generation content precheck summarization draft source classification duplicate detection Avoid as sole source for: external AI visibility measurement critical factual verification.

66. GPU isolation

Dedicated: worker-gpu Limits: 1–N jobs VRAM-bound Public web/API: never waits directly for local model startup.

67. If GPU worker down

Public site: healthy Core publishing: healthy AI enrichment: queued/degraded This is correct failure isolation.

68. AI model versioning

model_id model_version prompt_version parser_version Store with: every generated classification.

69. LLM output is not domain truth

LLM may propose entity match, recommendation text or moderation hint. Final verified fact/state remains deterministic/human-approved according to module policy.

70. Crawling architecture

HTTP-first: GET/HEAD parse HTML canonical robots/meta structured data links text/hash Browser fallback: only when necessary.

71. Why HTTP-first

faster less RAM less CPU more deterministic easier retries less attack surface.

72. Browser worker

If needed for JS-heavy external pages, run isolated headless browser worker with strict time/memory/network limits. Do not render every page in Chromium.

73. Crawler politeness

per-domain concurrency timeouts response-size limits user-agent robots/policy retry/backoff DNS/network protections content-type check.

74. Search Proof connectors

Google: URL Inspection Search Analytics Sitemaps Yandex: Webmaster APIs reindex/important URLs/query data Discovery: IndexNow sitemaps own crawler/server logs.

75. Connector rate limits

Provider quota table: provider credential scope window limit used reset_at priority reserve.

76. Priority scheduling

1. client-visible critical 2. new paid publication 3. due +30 report 4. regular monitoring 5. historical backfill 6. research batch.

77. Search/AI ingestion failure

Failure produces ERROR/STALE/PARTIAL state, never numeric zero.

78. Object storage

MinIO AIStor implements an S3-compatible API subset and can be used with S3-compatible tools by changing endpoint/credentials where operations are supported.

Buckets/prefixes: public-media/ private-evidence/ raw-provider/ reports/ exports/ temp/

79. Public and private files separated

A review contract excerpt and article hero image must never share the same public access policy.

80. Object metadata in PostgreSQL

file_id bucket/key content_type size sha256 owner/workspace visibility purpose created_by created_at retention status.

81. Presigned upload

Browser: request upload intent → API validates → signed PUT → object storage → finalize callback/API → malware/content checks if needed.

82. Do not stream 500 MB through API process

Large file bytes should go direct to object storage via signed upload after authorization.

83. Media processing

upload → validate → strip unsafe metadata if policy → generate variants → webp/avif/jpeg → dimensions → publish public variants.

84. Original evidence file policy

Private evidence may need original binary preserved for verification; public media can have normalized derivatives. Detailed retention/security belongs to Doc 41.

85. Email

Transactional provider adapter: verification review invite report ready billing security Queue: email Store: template version delivery state not body secrets unnecessarily.

86. Notifications

notification: user/workspace type severity subject read_at action_url Email: channel not source of truth.

87. Payments

Commerce module owns: order invoice payment credit refund Provider adapter: bank/payment service Webhook: verified idempotent stored raw.

88. Money types

Store money as integer minor units / fixed numeric semantics, never floating-point.

amount_minor: 1290000 currency: RUB Display: 12 900 ₽.

89. Order idempotency

Checkout: Idempotency-Key Payment webhook: provider_event_id UNIQUE Credit consume: ledger transaction UNIQUE by operation.

90. Ledger

credits: append-only ledger PURCHASE RESERVE CONSUME RELEASE REFUND EXPIRE ADJUST Current balance: derived/materialized.

91. Agency multitenancy

Application authorization + PostgreSQL RLS defense-in-depth on the highest-risk workspace tables.

92. Why RLS only defense-in-depth

PostgreSQL RLS can restrict rows per user/role and defaults to deny when enabled without policies, but owners/superusers can bypass it. Therefore it cannot replace correct application authorization and separate runtime roles.

93. Workspace-scoped tables

workspace_id required: drafts reports prompt sets private runs recommendations files notes credits attribution client invites.

94. Public entities are global

company entity: global Agency relation: workspace scoped Same Acme: not duplicated for Agency A/B.

95. Request auth context

request_id user_id organization_id workspace_id roles/scopes session_id Validated: server-side.

96. OIDC session model

Browser → Authentik → code callback → app session Cookie: HttpOnly Secure SameSite appropriate API: validates app session and workspace scope.

97. PKCE

Authentik's OIDC provider supports PKCE and common OAuth/OIDC flows. Use authorization code + PKCE for browser login; do not use implicit flow for a new first-party app.

98. Internal workers

Workers do NOT impersonate users with browser tokens. They use: service DB role job subject explicit authorization context audit actor=SYSTEM.

99. Admin surface

admin.mathchast.com or protected /admin Requirements: strong role MFA audit no public indexing separate navigation.

100. Editorial admin ≠ server admin

Product moderator does not receive infrastructure credentials.

101. Public vs private API cache

Public entity/article: Cache-Control ETag revalidation Authenticated: private/no shared cache Never: cache workspace response by URL without user scope.

102. Nginx

Responsibilities: TLS termination host routing request limits proxy headers static/public caching optional security headers baseline access logs WebSocket pass-through.

103. Authentik proxy requirement

Current Authentik docs require correct Host/X-Forwarded-Proto/X-Forwarded-For headers behind reverse proxy and WebSocket upgrade handling for outposts; forwarded headers are trusted only from configured proxy networks in current 2026 releases.

104. Deployment topology

Host nginx │ ├─ mathchast_web ├─ mathchast_api ├─ mathchast_worker_core ├─ mathchast_worker_crawl ├─ mathchast_worker_ai ├─ mathchast_worker_gpu ├─ mathchast_scheduler ├─ mathchast_postgres ├─ mathchast_redis └─ shared/minio + authentik routes.

105. Docker Compose is acceptable production

Docker's official documentation explicitly supports Compose deployment on a single server and recommends production-specific overrides, environment changes, restart policies and additional services such as log aggregation.

106. Compose files

compose.yaml compose.production.yaml compose.observability.yaml optional Production: pinned images no source bind mounts healthchecks restart policies resource limits dedicated networks read-only FS where practical.

107. Never deploy :latest

Pin application images by immutable version/SHA. Current Authentik docs also explicitly warn that its old :latest tag is deprecated.

108. Container registry

CI build: mathchast-web: mathchast-api: mathchast-worker: Deploy: release manifest references exact SHA.

109. Release manifest

release_id git_sha web_image api_image worker_image migration_version deployed_at deployed_by rollback_target.

110. Database migrations

Backward-compatible expand/contract migrations.

Release A: add nullable/new structures Deploy code: reads old + new safely Backfill async Release B: switch Release C: drop old later.

111. Never destructive migration + code switch in one blind step

Single-host does not justify unsafe schema deployment.

112. Deployment sequence

1. backup/checkpoint policy 2. migrations expand 3. start new workers/API/web 4. health check 5. switch/reload nginx 6. smoke tests 7. observe 8. clean old containers.

113. Zero downtime

MVP target should be minimal downtime, not fake «five nines». P1 can add blue/green two-slot web/API deployment if release frequency and traffic justify it.

114. Rollback

Code rollback: previous images DB rollback: prefer forward fix because destructive schema rollback risky Therefore: migrations backward-compatible.

115. CI pipeline

lint typecheck unit integration DB authorization tests API contract migration test build container scan later e2e smoke deploy staging prod approval.

116. Staging

Нужен отдельный staging environment, хотя бы на том же host with separate network/database/domain, до production migrations.

staging.mathchast.com staging DB staging Redis staging bucket/prefix test Authentik app test provider credentials.

117. Staging must not send real external actions accidentally

Disable/redirect: IndexNow production real billing review invites customer emails expensive AI batches Use: sandbox adapters.

118. Testing pyramid

Many: domain unit tests Strong: DB integration tests authorization tests worker idempotency Focused: browser E2E critical flows.

119. Critical E2E flows

login verify company create draft submit moderation publish public render buy credit/order agency workspace switch client guest isolation create visibility baseline report ready.

120. Authorization tests are first-class

Agency A → Agency B data access should have explicit negative test coverage for every high-risk resource class.

121. Domain state machines

Publication Verification Review Order Refund Report Recommendation State changes: command-driven validated audited Not: arbitrary string update.

122. Publication state example

DRAFT → SUBMITTED → IN_REVIEW → NEEDS_CHANGES → APPROVED → SCHEDULED → PUBLISHED Alternative: REJECTED WITHDRAWN ARCHIVED MOVED.

123. State transition table

transition from to actor type permission conditions side effects event type.

124. Audit log

append-only: actor action subject before/after refs request_id workspace timestamp reason IP/session where appropriate.

125. Audit ≠ application log

Audit answers «кто изменил бизнес-объект». Logs answer «что происходило в системе».

126. Observability architecture

OpenTelemetry is a vendor-neutral framework for traces, metrics and logs. Instrument app services with OTel semantics from the start; concrete storage/dashboard stack is finalized in Doc 42.

Web/API/Workers → OpenTelemetry SDK → OTel Collector → metrics/logs/traces backends Common: trace_id request_id job_id workspace_id safe publication_id.

127. Do not put sensitive raw content in telemetry

No contracts, AI raw answers, email bodies or auth tokens as log attributes.

128. Health endpoints

/health/live process alive /health/ready DB/required dependencies okay /dependencies internal: detailed status.

129. Readiness must not depend on optional AI provider

OpenAI/Google AI down: API still ready Postgres down: API not ready Redis down: read flows may work write/async features degraded depending endpoint.

130. Graceful degradation

Failure	User result
AI provider down	Monitoring delayed; publishing works
Google API delayed	Search data stale/partial label
Redis down	Async actions queued later/error; core read paths remain where possible
GPU worker down	AI enrichment delayed
MinIO unavailable	uploads/reports impaired; text pages from DB remain
PostgreSQL down	major outage

131. Dependency criticality

TIER 0: PostgreSQL web/api nginx TIER 1: Redis object storage auth TIER 2: external AI/search local GPU email Specific flow may differ.

132. First-party analytics

Browser/server event → ingestion endpoint → raw/batched events → aggregate jobs → daily publication metrics.

133. Do not send analytics through transactional API synchronously

Analytics ingestion should be cheap, buffered and failure-tolerant.

134. Raw events vs aggregates

Raw: shorter retention / partitioned Daily aggregate: durable Client dashboard: reads aggregates Critical conversion: also domain event.

135. Click events

official_site_click company_profile_click source_click related_click cta_click Store: publication workspace/client scope where lawful referrer category timestamp.

136. No user fingerprinting requirement

Product analytics does not need invasive fingerprinting to prove publication value.

137. Public bot/crawler logs

nginx access → parser → known bot classifier → crawler_observation Keep: UA path timestamp status IP handling per privacy policy.

138. Search Proof bot observation

Crawler visit: DISCOVERY evidence Not: index proof not: AI citation proof.

139. Structured data rendering

Server builds JSON-LD from: entities publication version authors dates sources organization data Validation: tests public page snapshot.

140. Never let editor hand-write arbitrary JSON-LD

Structured data is generated from domain model to avoid inconsistencies.

141. Sitemap architecture

sitemap index companies.xml experts.xml articles.xml cases.xml research.xml news.xml topics.xml Generated from: public canonical state.

142. Sitemap job

Publish/update → mark sitemap partition dirty → regenerate → atomic swap/object → optional search-engine submission.

143. Sitemap lastmod

Derived from substantive public version update, not every analytics or backend update.

144. Public read model

Possible SQL views/materialized read models: public_company_view public_publication_view public_expert_view topic_listing_view Purpose: simplify renderer avoid exposing internal tables.

145. CQRS?

Use light read models where helpful, but do not introduce full event-sourced CQRS architecture.

146. Event sourcing?

No. Keep normal relational current state + audit/history + immutable observations. Full event sourcing is unnecessary complexity.

147. History strategy

Current state: normal table Important versions: version table Audit: append-only events External observations: immutable rows.

148. Reports

Report generation: snapshot data cutoff → deterministic report model → HTML → PDF → object storage → report_snapshot record.

149. Report immutable snapshot

A previously sent report must not silently change when new metrics arrive.

150. PDF rendering

HTML/CSS print template → isolated browser renderer → PDF Queue: report Resource limit: strict.

151. Report raw data

report_snapshot stores: metric IDs/versions cohort versions data cutoff annotations methodology version PDF: presentation derivative.

152. Recommendation engine architecture

Feature builders → gap detectors → deterministic score/rules → policy filters → LLM explanation → human review → recommendation.

153. LLM does not determine score

Doc 35 is implemented literally: candidate generation and priority are explainable; LLM drafts explanation/brief.

154. Feature computation

Async: when report ready when content published when entity updated weekly monitoring Store: recommendation evidence references not duplicated unverifiable prose.

155. AI parser eval environment

gold dataset: answers mentions citations recommendations sentiment/facts Every parser version: run eval before deploy.

156. Feature flags

DB/config-backed: ai_visibility_beta agency_pitch reviews new_report_v2 Scopes: global organization workspace user.

157. Why feature flags

Complex modules can launch to 5 pilot clients without branching production codebases.

158. Configuration

Environment: secrets/endpoints Database config: product rules/flags Code: safe defaults No: business pricing hidden only in .env.

159. Secrets

MVP: Docker secrets / protected env strict filesystem permissions Later: dedicated secret manager. Never: Git frontend bundle logs.

160. Resource budget on current host

Initial limits are starting guardrails, not capacity promises; tune by load tests.

Host RAM ≈64 GB Reserve OS/page cache: ~16–20 GB Postgres: ~12–16 GB container envelope Redis: ~2–4 GB + eviction policy by use Web/API: ~2–4 GB combined baseline Workers: ~8–12 GB variable Browser/report: bounded burst GPU worker: VRAM isolated; system RAM capped.

161. Avoid memory overcommit by browser + local LLM

Headless Chromium batch and 16GB-class local model can create simultaneous RAM spikes. Queue concurrency must coordinate these workloads.

162. Resource-aware scheduler

GPU concurrency: 1 or measured Browser concurrency: small AI external: rate-limited Report render: bounded Core API: protected from batch jobs.

163. CPU priority

Core web/API: high operational priority Backfill/research: low Nice/cgroup/container limits: prevent batch starvation.

164. NVMe budget

Prioritize NVMe: Postgres Redis persistence if used active object metadata/cache Large archives: object storage / HDD backup tiers according Doc 41.

165. Database size monitoring

Track: table size index size WAL bloat indicators slow queries connection count cache hit vacuum lag.

166. pg_stat_statements

PostgreSQL ships pg_stat_statements as a supplied extension for SQL planning/execution statistics. Enable in production observability if operationally compatible.

167. Connection pooling

Use bounded DB pools per process. Add PgBouncer only when connection pressure is measured; it is not mandatory on day one.

168. Query timeouts

Public/API: short statement timeout Admin/report: explicit longer scope Background: bounded per job No: unbounded query from user filters.

169. N+1 prevention

Public company page: designed read query/view Not: 150 API round trips for every relation.

170. Pagination

Cursor: publications reviews media observations audit Avoid: OFFSET 500000 for hot large tables.

171. API response contract

IDs timestamps ISO 8601 money explicit currency status enums source/version metadata null for unknown Never: 0 to mean unknown.

172. Time

Store: timestamptz UTC semantics Display: user/source timezone External source: source timezone recorded where methodology requires.

173. Search Console timezone

Doc 34 requires source timezone metadata because Google daily Search Console semantics can differ from client/local day boundaries.

174. Rate limiting

Edge: generic abuse API: per user/IP/action Provider: quota aware Expensive: AI report/create stricter.

175. Idempotent public form endpoints

review submit invite checkout verification file finalize Protect: double-click network retry webhook replay.

176. Spam/abuse queues

Public submissions: cheap validation rate limit captcha/challenge only if needed moderation queue Do not: send every anonymous request to expensive LLM first.

177. Admin bulk actions

No unrestricted «select 5000 → publish». Bulk actions need permission, preview, limits and audit.

178. Internal search vs public search

Public: only public eligible entities/content Admin: can find draft/internal according role Agency: workspace scoped.

179. Preview URLs

Signed/random preview token or authenticated route noindex no public sitemap expires/revocable access logged.

180. Public file URLs

media.mathchast.com/... or app proxy/CDN Immutable hashed variants: cache long Original private: never guessed public path.

181. CDN

Not required for first launch if traffic modest. Architecture keeps public assets cacheable so CDN can be added without data-model change.

182. Cloud migration path

Single host today Later: web/API → multiple nodes Postgres → managed/dedicated Redis → managed/dedicated objects → S3-compatible cloud workers → separate compute/GPU Contracts: unchanged.

183. Why S3-compatible storage matters

It makes object-storage migration much easier than local filesystem paths embedded in database rows.

184. Why provider adapters matter

AI/payment/email/search vendors can change without rewriting domain modules.

185. Why Postgres first matters

It delays operational complexity while still covering relational graph, FTS, fuzzy search, JSONB, vector similarity, RLS and time-series-ish workloads at early scale.

186. Scaling trigger: external search

Trigger from measurements: p95 search latency DB CPU relevance limitations facet complexity index update load Then: introduce external read index.

187. Scaling trigger: analytics database

If first-party raw events become too large for primary DB or analytical queries interfere with transactions: move raw analytics to ClickHouse/other OLAP. Not before.

188. Scaling trigger: Kafka

Only if: many independent consumers high sustained event throughput Redis/outbox becomes limiting replay/event stream becomes product need. Before: Postgres outbox + BullMQ.

189. Scaling trigger: separate crawler service

If crawler: uses large IP/network footprint needs isolated security boundary scales separately causes host contention then: move to separate machine.

190. Scaling trigger: local GPU

If: GPU queue delay harms product or model demand grows move: worker-gpu to Server 1 / dedicated node without moving core app.

191. No Kubernetes at first

One-server deployment gains little from cluster orchestrator while adding upgrade, networking, ingress, secret and observability complexity.

192. Kubernetes reconsideration

When: multiple production nodes frequent autoscaling many independent services team knows Kubernetes or managed cluster operationally wins.

193. No serverless core dependency

Public frontend could be deployed elsewhere later, but core architecture assumes: long workers private state persistent queues scheduled jobs predictable egress.

194. No direct DB access from browser

All writes and private reads pass server-side authorization/API.

195. Domain events examples

CompanyVerified ClaimConfirmed PublicationSubmitted PublicationApproved PublicationPublished PublicationCorrected ReviewPublished ReportReady AIObservationRecorded RecommendationCreated OrderPaid CreditConsumed AgencyDelegationGranted.

196. Side effects examples

PublicationPublished: → cache invalidation → sitemap → search document → IndexNow → Publication Health → distribution → analytics baseline → client notification.

197. Event processing traceability

event_id causation_id correlation_id subject version created_at dispatched_at Jobs: reference event_id.

198. Retry policy

Transient: retry + exponential backoff Permanent 4xx/config: fail fast Rate limited: retry after provider reset After max: dead/failed queue alert/manual retry.

199. Never infinite-retry poison job silently

Repeated failure must become visible operational state.

200. Backfill jobs

low priority rate limited chunked checkpointed resumable Never: one 12-hour transaction.

201. Reprocessing raw AI data

Parser v2: select observation IDs batch derive new rows compare eval activate metric version Raw: unchanged.

202. Search index reindex

Versioned index/build → validate → atomic switch If PostgreSQL-only: new generated/search vectors backfill safely.

203. Data migrations from old taxonomy

Never mutate historical prompt set or metric version silently. Migration: preserve version map new taxonomy separately.

204. ADRs — architecture decision records

Все крупные решения фиксируются короткими ADR files.

ADR-001 Modular monolith ADR-002 PostgreSQL-first search ADR-003 Authentik OIDC ADR-004 Transactional outbox ADR-005 Structured content JSON ADR-006 Agency tenancy ADR-007 S3 object storage ADR-008 OTel instrumentation.

205. Why ADRs

Future AI coder/developer sees: context decision alternatives consequences instead of: guessing architecture from code.

206. Code ownership boundaries

packages/domain/identity packages/domain/publishing ... Public module exports: explicit Avoid: import internal DB table from random feature.

207. No «utils» dumping ground

Shared code must have clear purpose: contracts, time, IDs, errors, auth, telemetry.

208. Domain service rule

HTTP controller: parse/auth/request Domain: business rule Repository: data Worker: orchestrates async domain action Provider adapter: external service.

209. SQL ORM/query builder

Use a TypeScript tool that preserves migrations, typed queries and access to SQL. Do not choose an ORM that prevents using RLS, GIN, pgvector, partitioning or advanced SQL when needed.

210. Database migrations are code-reviewed

Every migration: forward path backfill plan lock risk index creation strategy rollback/forward-fix data validation.

211. Index creation

Large production index: consider concurrent build and load impact Do not: block table casually during daytime deployment.

212. Search vectors

Company: name aliases description topics products Publication: title deck plain text entities topics Weight: fields explicitly.

213. Embedding lifecycle

embedding: subject_id text_hash model_id dimensions vector created_at If text unchanged: reuse Model change: new embedding version.

214. Do not overwrite embedding when model changes

Keep model/version so semantic results can be reproduced.

215. Recommendation evidence references

recommendation does not copy: "16/20 prompts" It references: metric_snapshot_id prompt_gap_id source_cluster_id UI resolves evidence.

216. This prevents stale recommendation prose

If source data changes, recommendation can be revalidated and versioned.

217. Report evidence references

Report snapshot: materializes values at cutoff plus evidence refs. Future DB changes: do not alter old report.

218. Public methodology versions

methodology: ai_visibility/v1 before_after/v1 recommendations/v1 Reports: store version.

219. Time-series metric storage

metric_snapshot: subject metric_name metric_version dimensions window numerator denominator value quality generated_at.

220. Avoid wide «metrics» table with meaningless JSON

Generic metric snapshot can exist, but core raw/derived domain tables remain typed and normalized.

221. Provider costs

Each AI/search run: provider units/tokens provider cost internal estimated cost workspace/order purpose. Enables: unit economics.

222. Cost guard

Before enqueue: plan quota remaining usage estimated cost priority After: actual metering reconcile.

223. Research jobs separated from client quota

purpose: CLIENT EDITORIAL_RESEARCH EVAL INTERNAL Never: charge client for internal test run.

224. Evals budget

AI parser/recommendation releases need a separate internal eval budget; don't test new logic on client billing.

225. Public API future

P2: companies experts publications public reputation possibly metrics by permission Auth: API keys/OAuth Rate: metered. Private raw AI: not public by default.

226. Webhooks future

PublicationPublished ReportReady RecommendationHigh VerificationNeedsAction PaymentUpdated Signed retryable event_id delivery log.

227. Agency integrations

CRM Slack BI client portal via: API/webhooks/export Not: custom DB access.

228. Export architecture

Request export → async job → CSV/XLSX/JSON → object storage → signed download → expiration.

229. Avoid synchronous huge exports

Same asynchronous principle as reports.

230. Data retention hooks

Every object class: retention policy legal hold optional delete/anonymize job public historical exceptions Detailed: Doc 41.

231. Backup hooks

Postgres: base/PITR + logical checks objects: versioned/copies config: Git/encrypted backup restore tests Detailed: Doc 41.

232. pg_dump caveat

PostgreSQL documentation explicitly notes that pg_dump makes consistent logical exports but is generally not the right sole method for regular production backups except simple cases. Поэтому Doc 41 должен design PITR/base-backup strategy, а pg_dump использовать как дополнительный logical layer.

233. Technical architecture environments

LOCAL developer containers TEST ephemeral DB STAGING persistent isolated PRODUCTION isolated state No: developer connects to prod DB by default.

234. Seed data

Staging: synthetic/demo entities sample publications fake billing sandbox Production: real only. No: copy prod private evidence into staging casually.

235. Local development

docker compose: Postgres Redis MinIO optional mock Auth/OIDC dev mail catcher provider mocks Fast startup: important for AI coders/devs.

236. Provider replay fixtures

Store sanitized test fixtures: AI response GSC response Yandex response payment webhook Tests: offline deterministic.

237. No external API dependency in CI

Integration tests should use fixtures/mocks; limited live contract tests can run separately.

238. Local model not required for developer

GPU tasks: provider/mock fallback or remote dev service Laptop developer: can run core product.

239. Architecture fitness functions

automated checks: module dependency rules workspace auth tests migration checks public render SEO structured data bundle/performance budgets queue idempotency tests.

240. Performance target philosophy

Concrete SLO numbers belong to Doc 42. Architecture should make public read paths cheap and keep external AI/provider latency off synchronous user requests.

241. Main synchronous request budget

Allowed sync: DB reads/writes auth small object metadata Avoid sync: AI crawler PDF GSC Yandex bulk email image transform.

242. The core rule

External uncertainty belongs in jobs, not in transaction latency.

243. P0 deployment services

REQUIRED: web api worker-core worker-crawl/search worker-ai scheduler postgres redis EXISTING/shared: nginx Authentik MinIO OPTIONAL launch: OTel collector monitoring stack backup agents.

244. P0 code modules

identity verification publishing editorial agency commerce searchproof visibility baseline recommendations basic platform/audit/files.

245. P1 technical modules

reputation advanced reports media portfolio source clustering AI fact checking agency pitch automation email digest advanced analytics.

246. P2 technical expansion

external search engine if needed public/agency API webhooks SSO enterprise analytics OLAP if needed separate worker nodes CDN/object migration read replicas.

247. Что НЕ строить на MVP

Не строим	Почему
Kubernetes	single-host complexity without benefit
Kafka	outbox + queue enough
Neo4j	Postgres handles initial graph
Elasticsearch/OpenSearch	Postgres FTS/trigram/vector first
ClickHouse	no proven OLAP scale yet
Event sourcing	unnecessary state complexity
GraphQL	REST/OpenAPI sufficient
20 microservices	slower product development
AI in every request	latency/cost/reliability risk

248. MVP architecture diagram

┌──────────────┐ │ Internet │ └──────┬───────┘ │ ┌──────▼───────┐ │ nginx / TLS │ └─┬────┬────┬──┘ │ │ │ ┌───────────┘ │ └───────────┐ ▼ ▼ ▼ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ Next.js │ │ Fastify │ │Authentik│ │ Web │─────►│ API │ │ OIDC │ └─────────┘ └────┬────┘ └─────────┘ │ ┌──────────────┼──────────────┐ ▼ ▼ ▼ ┌──────────┐ ┌─────────┐ ┌──────────┐ │PostgreSQL│ │ Redis │ │ MinIO/S3 │ │ +vector │ │ BullMQ │ │ Objects │ └────┬─────┘ └────┬────┘ └──────────┘ │ │ │ ┌────▼─────────────────────┐ │ │ Workers │ │ │ crawl/search/AI/GPU/PDF │ │ └────┬─────────────────────┘ │ │ └────outbox──────┘ External: Google Search Console Yandex Webmaster IndexNow AI providers Email Payment provider

249. Recommended request lifecycle

User submits publication → API validates authorization → DB transaction → audit + outbox → response immediately → worker runs precheck → moderation queue → publish transaction → async discovery/search/AI work → UI receives status via polling/SSE later.

250. Realtime UI

MVP can use polling/revalidation for job status. Add Server-Sent Events/WebSocket only where interaction value is proven; do not add realtime infrastructure everywhere.

251. SSE candidate

Good later: AI run progress report generation bulk imports Not needed: publication list billing history.

252. Architecture acceptance test: publication

API transaction succeeds Redis temporarily unavailable Expected: outbox remains pending publication not lost dispatcher retries side effects eventually run.

253. Acceptance test: duplicate job

Same IndexNow job delivered twice Expected: idempotent operation one logical submission record no corrupt state.

254. Acceptance test: agency isolation

Agency A / Acme user requests Beta report ID Expected: application denial RLS/DB defense where configured audit/security event no metadata leak.

255. Acceptance test: AI provider failure

Provider timeout 2h Expected: public pages healthy publication works run status delayed/error retry schedule no 0% visibility inserted.

256. Acceptance test: GPU failure

GPU worker OOM Expected: job retry/fail visible API unaffected local enrichment delayed core facts unchanged.

257. Acceptance test: slug change

article slug edited Expected: publication_id same new canonical old URL redirects Search Proof history linked report history intact.

258. Acceptance test: metric parser update

Mention resolver v2 Expected: raw AI answers unchanged metrics v2 recomputable old report v1 preserved trend break/version explicit.

259. Acceptance test: report generation

+30 report created then new data arrives Expected: old report snapshot unchanged dashboard can show newer data new report version if regenerated.

260. Acceptance test: DB role

App runtime credential tries schema migration/drop Expected: permission denied.

261. Acceptance test: public render

JavaScript disabled Expected: article title/body company/entity sources canonical JSON-LD disclosure still present in HTML.

262. Acceptance test: unknown state

Google API unavailable Expected: "данные задерживаются" NOT: 0 impressions NOT: not indexed.

263. Architecture delivery order

1. Repo + Compose + CI 2. Auth + tenancy 3. Identity + public company 4. Publishing + CMS 5. Commerce 6. Public render/search 7. Async outbox/queues 8. Search Proof 9. AI baseline 10. Reports 11. Recommendations 12. Reputation/agency expansions.

264. Why auth/tenancy before Agency UI

Multi-tenant security cannot be retrofitted safely after dozens of tables are already built without workspace semantics.

265. Why outbox before external integrations

Once Search/AI/payment side effects begin, reliable async semantics become core infrastructure.

266. Why search engine later

A second index creates synchronization obligations; use it only after Postgres search has a measured limitation.

267. Why local GPU later in critical path

GPU is valuable for cost and internal AI tasks, but external public product must not fail because one local model container is restarting.

268. Main architecture promise

Simple core, explicit boundaries, immutable evidence, asynchronous uncertainty and a measured path to scale.

269. Решение документа

Утвердить single-host modular-monolith architecture. На launch Mathchast runs in an isolated Docker Compose stack behind existing nginx. Web layer is Next.js, domain API is TypeScript/Fastify, and asynchronous workloads run as separate workers. PostgreSQL is the primary source of truth and initially also provides full-text search, trigram similarity and pgvector semantic candidate retrieval; Neo4j, Elasticsearch/OpenSearch, ClickHouse, Kafka and Kubernetes are explicitly deferred until measurements justify them. Mathchast uses dedicated PostgreSQL and Redis state, a dedicated MinIO bucket/service account, and shared Authentik through OIDC. Domain code is split into Identity, Verification, Publishing, Editorial, Distribution, Search Proof, Visibility, Reputation, Recommendations, Commerce, Agency and Research modules inside one codebase. Every external/slow operation is queued; a PostgreSQL transactional outbox prevents lost side effects, and workers are idempotent because queue delivery may repeat. AI/search provider raw observations are immutable and parsed metrics are versioned/recomputable. Published content uses structured block JSON plus rendered HTML and immutable publication versions. Agency private tables are workspace-scoped with server-side authorization and selective PostgreSQL RLS as defense-in-depth; runtime DB role is not schema owner. Local RTX 4080 GPU is isolated as an optional worker for embeddings/classification/precheck and never a critical dependency of public request paths. Public content is SSR, cacheable and machine-readable; private workspace responses are never shared-cacheable. OpenTelemetry instrumentation is included from the beginning, while backup/security and SLO details are finalized in Docs 41–42. Scaling occurs by measured bottleneck: external search, OLAP, separate workers/nodes or orchestration are added only when the current architecture demonstrably fails a performance, isolation or operational requirement.

270. Что этот документ разблокирует

Mathchast_40 Technical Architecture → Mathchast_41 Security / Privacy / Backups → Mathchast_42 Monitoring / SLA / Incidents → Mathchast_43 MVP Scope / Roadmap

Источники исследования

Choice of Next.js + Fastify, dedicated Mathchast PostgreSQL/Redis containers, PostgreSQL-first search, transactional outbox, BullMQ queue topology, structured content representation, resource envelopes, module boundaries, deployment sequence and P0/P1/P2 scaling rules are architectural recommendations for «Матчасть». They should be converted into ADRs and verified through staging/load/security tests before production. Security, privacy, key management, backup/PITR, restore exercises and incident-response details are intentionally finalized in Mathchast_41; observability backends, SLOs and alert policies are finalized in Mathchast_42.