AI engineer · model compression, RAG, agents.
I build a small open-source AI stack: quantize → retrieve → reason.
Three pip-installable libraries, all live on PyPI.
I'm building a small, readable, fully-open-source AI stack around one question: how do we take a large open model, shrink it, serve it locally, and make it actually reason about your data — without an API key?
1. TurboQuantShrink the model. INT4 · INT8 · FP16 · GPTQ · AWQ · BitsAndBytes · structured pruning · ONNX / TensorRT. pip install turboquant-mlfrom turboquant import quantize
qmodel = quantize(model, method="bnb-nf4")
|
2. RAGforgeRetrieve over your docs.
PDF / Markdown ingestion · Qdrant (embedded) · bge reranking · pip install ragforge-mlfrom ragforge import Pipeline
rag = Pipeline.from_defaults(model_id="...")
rag.ingest(["docs/"])
rag.ask("...")
|
3. AgentForgeReason and use tools.
ReAct loop · calculator · sandboxed Python REPL · web search · SQL · RAG · pip install agentforge-mlfrom agentforge import Agent
agent = Agent.from_defaults(
model_id="...",
tools=[Calculator(), RAGTool(rag)],
)
agent.run("...")
|
The three compose: an agentforge.Agent uses an agentforge.tools.RAGTool
that wraps a ragforge.Pipeline, both running on an agentforge.llm.QuantizedHFLLM
that delegates to turboquant.quantize(method="bnb-nf4"). Open models, local
indexes, no API key required.
| Area | Tools |
|---|---|
| Model compression | PyTorch · BitsAndBytes · GPTQ · AWQ · ONNX · TensorRT |
| RAG / retrieval | sentence-transformers · BGE · Qdrant · FastAPI |
| Agents | ReAct · tool use · sandboxed exec · SQLite memory |
| LLMs | HuggingFace Transformers · Llama · Qwen · Mistral · SmolLM |
| Tooling | GitHub Actions · MkDocs Material · pytest · ruff · twine |
| Languages | Python (primary) · C / C++ · Lua |
- Shrinking large open models so they run on the hardware people actually have.
- Local-first, no-API-key tooling — projects you can clone, install, and reproduce in 5 minutes.
- Measurable — every change should answer "did it get better?", not "does it run?".
- Readable algorithms — each technique is one short module that doubles as a reference for how it works.
Open an issue on any of the three repos — or DM via GitHub.