Title: AI Retrieval & Agent Platform Engineer

Position Summary: 

Join us to build the retrieval and agent connectivity layer of our Smart Data Fabric. You’ll design hybrid retrieval pipelines (graph + vector + business logic), optimize vector databases, implement RAG, and deliver MCP-based tool wrappers for enterprise systems. You’ll also own observability for agent workflows—latency, accuracy, relevance, and cost—creating automated feedback loops that continuously improve performance.

How You’ll Make an Impact (responsibilities of role)

Vector DB & Hybrid Retrieval

• Stand up and tune vector databases (Pinecone/Weaviate/Qdrant/AWS-native) for similarity search at scale.
• Design hybrid retrieval combining vector semantic search with graph context and business logic filters; implement re-ranking.
• Manage embedding lifecycle (choice, diversity, refresh cadence, cold-start strategies).

RAG & Contextualization

• Build RAG pipelines pulling structured/unstructured context; implement chunking, metadata, and guardrails.

• Integrate graph-derived context windows for multi-hop reasoning in agent workflows.

Agent Connectivity (MCP) & Tooling

• Implement MCP-based tool discovery/invocation for agent ↔ system interaction.
• Wrap enterprise systems (Snowflake/MongoDB/SharePoint/ERP) as reusable tools/skills with clear schemas/capabilities.

• Represent tool capabilities & dependencies as graph processes for orchestration; collaborate with graph team.

Observability & Feedback Loops

• Instrument agent KPIs (latency, accuracy, relevance, cost/execution); implement tracing across retrieval/graph layers.
• Build dashboards and automated feedback loops (e.g., low relevance → retraining/embedding refresh; failures → rule updates).

• Optimize cloud architecture for performance, cost, security; maintain SLOs.

Cloud & Performance

• Deploy and scale retrieval services, vector stores, and agent endpoints on AWS (IAM, VPC, S3, Lambda, EKS/ECS, DynamoDB).

• Conduct performance profiling, caching strategies, and cost optimization (e.g., batch upserts, ANN index selection, sharding).

What You Bring (required qualification and skill sets)

• Bachelor’s/Master’s in CS, Data Science, Engineering, or related field.
• 6–10 years in IR/Retrieval systems, vector DBs, or agent platform engineering.
• Hands-on with Pinecone/Weaviate/Qdrant (at least one in production), embeddings, ANN indexes, and hybrid ranking.
• Experience building RAG pipelines and contextualization strategies with LLMs.
• Strong API/backend engineering skills (Python/TypeScript/Java) for tool wrappers and retrieval services.
• Practical AWS experience (IAM, VPC, S3, Lambda, EKS/ECS, DynamoDB), security-by-design.

• Familiarity with graph queries (Cypher/Gremlin/SPARQL) to leverage semantic context.

Preferred Qualifications

• Experience with MCP or equivalent agent–tool interoperability patterns; skill registries and capability discovery.
• Observability stack: OpenTelemetry, Prometheus/Grafana, distributed tracing; KPI-driven optimization.
• Knowledge of LangChain/LlamaIndex, vector re-ranking, prompt caching, and safety/guardrail mechanisms.

• Exposure to Neo4j/Neptune/TigerGraph; event streaming (Kafka/Kinesis) for ingestion/update triggers.