ensia-impact-chatbot

ENSIA Impact Chatbot Building a chatbot that answers questions about content shared in the ENSIA IMPACT Telegram supergroup (18 topic channels: Startups, Research, Opportunities, Companies, Q&A, etc.). Project structure Setup The chatbot uses LiteLLM so swapping LLM providers is a one-line change in .env. Default is groq/llama-3.3-70b-versatile. Pipeline Single command The script chains all four stages and is safe to re-run anytime — perfect for picking up new data: Stage When it does work When it skips --- --- --- 1. Extract PDFs Always (fast, 5s) — 2. OCR photos & scanned PDFs New photos / new scanned PDFs only Anything already OCR'd 3. Merge OCR into chat Always; rewrites file only if content changed — 4. Scrape ensia.edu.dz Re-fetches only pages whose modified timestamp changed Unchanged pages skipped via the manifest 5. Scrape v2v.ensia.edu.dz Always re-renders (single-page app, no modified signal) — 6. Build retrieval index Adds new chunks; full rebuild if any source data is newer than the index Re-runs are no-ops when nothing changed Manual / per-stage runs Data summary After the full pipeline: Source Chars Notes --- --- ---