Architecture: helvia-rag-pipelines

C4 Component Diagram

Key Flows

Process Query Flow

1. POST /pipelines/{pipeline_id}:process received; JWTBearer validates token and pipeline_id claim.
2. PipelineService.process() is called.
3. If translation.enabled and query language differs from pipeline's native languages, query is translated via TranslationService.
4. Previous messages are loaded from MessageHistoryService (if chat_history.enabled and session_id provided).
5. If query_summarization.enabled and history exists, LLMService.summarize_user_query() rewrites the query in context.
6. If semantic_search.enabled, SemanticSearchService.search() embeds the query (with optional cache) and searches the vector DB for top-k results.
7. LLMService.process() constructs a prompt from the corpus results and calls the configured LLM client; response is parsed by TextGenerationParser.
8. If response language differs from query language, TranslationService.translate_response() translates back.
9. Chat history is stored; LLM token count is recorded in MySQL.
10. Optional: if sem_cache.enabled, result is stored in SemCacheService for future semantic cache hits.

Training / Indexing Flow

1. POST /pipelines/{pipeline_id}:train sets pipeline status to TRAINING.
2. PipelineService.train() calls _index_corpus().
3. SemanticSearchService.index_corpus() fetches all corpus items with need_training=True (or all if force_reindex=True).
4. For each item, calls the configured embedding client to generate a vector.
5. Vectors are upserted into the vector DB collection via VectorDbManager.
6. Pipeline status is set to READY; last_trained_at is updated; corpus items are marked trained.

Corpus Update Flow

1. PUT /pipelines/{pipeline_id}/corpus receives the full desired corpus.
2. PipelineService.update_corpus() diffs incoming items against existing MySQL corpus (insert/update/delete).
3. For new or changed items, _prepare_training_text() formats the article text (using article_format template with {{title}}, {{group}}, {{body}}, {{tags}}).
4. If the corpus language differs from the pipeline's native language, items are translated before storage.
5. Pipeline status is set to OUTDATED (requires re-training).