<div>Umanist Staffing LLC A Small Business Enterprise (SBE); as per the gov of the USA, is the global advisor on niche skills workforce solutions. </div>

Umanist Staffing LLC

# AI Solutions Engineer **Location:** Pune, Maharashtra **Work Mode:** Work From Office (WFO)(890) **Experience:** 9–12 Years&nbsp; **Notice Period:** Immediate to 45 Days **Preferred Candidates:** Local candidates based in Pune
## Role Overview
We are seeking an experienced AI Solutions Engineer to develop, evaluate, test, monitor, and support AI agents and LLM-powered applications across their development and production lifecycle.
The ideal candidate must have strong hands-on expertise in Python, LangChain, LangGraph, OpenAI, and Azure Databricks. The role involves agent orchestration, LLM evaluation, experiment lifecycle management, automated testing, observability, production monitoring, troubleshooting, and operational reliability.
This is a client-facing position involving regular interaction with clients and technical teams. Excellent verbal and written communication skills are mandatory, with no compromise on communication quality.
## Key Responsibilities
### 1. Agent Development &amp; Orchestration - Design, develop, integrate, and maintain AI agents using LangChain, LangGraph, and OpenAI-based technologies. - Build Python-based services, REST APIs, and agent integration components using FastAPI and Pydantic. - Work with agent orchestration frameworks such as OpenAI Agents SDK, Google ADK, and Pydantic-AI, where applicable. - Support agent workflow development, integration, testing, and production operations.
### 2. LLM Evaluation &amp; Experiment Lifecycle - Design and implement evaluation frameworks to assess agent quality, response accuracy, reliability, consistency, and performance. - Develop automated evaluation suites using pytest and custom pytest-based evaluations. - Work with MLflow Evaluation, DeepEval, Ragas, and promptfoo for LLM and agent evaluation. - Manage experiment execution, evaluation results, test iterations, and quality improvement cycles. - Identify performance gaps and support continuous improvement of AI agent outputs.
### 3. Agent Observability &amp; Tracing - Implement tracing, monitoring, and observability for agent workflows and LLM applications. - Use OpenTelemetry and OpenInference for telemetry and instrumentation. - Work with Langfuse and LangSmith to trace agent execution, investigate failures, and analyze application behavior. - Monitor agent performance and troubleshoot issues across application and integration layers.
### 4. Azure AI Stack - Work with Azure OpenAI and Microsoft Foundry for AI application and agent-related workloads. - Use Azure Monitor and Application Insights for telemetry, application monitoring, diagnostics, and troubleshooting. - Support operational monitoring and performance analysis of Azure-hosted AI solutions.
### 5. Azure Databricks &amp; Data Operations - Work with Azure Databricks for data processing, SQL-based analysis, and operational workflows. - Use Unity Catalog for data governance and access management. - Work with Databricks Workflows/Jobs and SQL Warehouses. - Support data-related workflows required for AI agent evaluation and operations.
### 6. CI/CD, Testing &amp; Version Control - Maintain source code and version control using Git and GitLab. - Integrate automated tests and evaluation suites into CI/CD pipelines. - Support repeatable testing, deployment validation, and release quality checks. - Troubleshoot build, test, and deployment issues.
### 7. Production Monitoring &amp; Reliability - Monitor applications and services using Grafana, Prometheus, Azure Monitor, Kibana, and alerting platforms. - Investigate production incidents, analyze logs and telemetry, and perform root-cause analysis. - Support incident response, issue resolution, performance monitoring, and operational improvements. - Help improve system reliability, observability, and troubleshooting efficiency.
### 8. Cloud Infrastructure, Containers &amp; Security - Work with Docker, Kubernetes, and Azure Kubernetes Service (AKS) for containerized workloads. - Understand infrastructure-as-code practices using Terraform and Bicep. - Apply security and identity management practices using Microsoft Entra ID, managed identities, OAuth2, Azure Key Vault, and role-based access control (RBAC). - Support secure application access, credential management, and appropriate authorization controls.
### 9. Client &amp; Stakeholder Collaboration - Collaborate regularly with clients, engineering teams, and other stakeholders. - Communicate technical issues, evaluation outcomes, incident updates, and resolutions clearly. - Participate in technical discussions and troubleshooting sessions. - Maintain professional communication and timely coordination across teams.
## Mandatory Skills / Must-Haves
The following skills are explicitly mandatory for this position:
1. Python 2. LangChain 3. LangGraph 4. OpenAI 5. Azure Databricks
Candidates must also demonstrate excellent communication, analytical thinking, problem-solving, and troubleshooting capabilities appropriate for a client-facing engineering role.
## Nice-to-Have Skills
### Programming &amp; APIs - FastAPI - REST APIs - Pydantic - pytest
### Agent Orchestration - OpenAI Agents SDK - Google ADK - Pydantic-AI
### Agent Observability - OpenTelemetry - OpenInference - Langfuse - LangSmith
### LLM Evaluation &amp; Experiment Lifecycle - MLflow Evaluation - DeepEval - Ragas - promptfoo - Custom pytest evaluations
### Azure AI Stack - Azure OpenAI - Microsoft Foundry - Azure Monitor - Application Insights
### Databricks - Unity Catalog - Databricks Workflows/Jobs - SQL Warehouses
### CI/CD &amp; Version Control - Git - GitLab - Automated testing pipelines
### Production Monitoring - Grafana - Prometheus - Azure Monitor - Kibana - Alerting platforms
### Cloud &amp; Containers - Docker - Kubernetes - Azure Kubernetes Service (AKS) - Terraform - Bicep
### Security &amp; Identity - Microsoft Entra ID - Managed identities - OAuth2 - Key Vault - Role-Based Access Control (RBAC)
## Candidate Screening Criteria
- **Experience:** 9–12 years(relevnat should be 8+yrs). - **Employment stability:** Minimum 2 years in an organization. - **Notice period:** Immediate to 45 days. - **Communication:** Excellent verbal and written communication is mandatory; no compromise. - **Location:** Local Pune candidates are preferred. - **Work arrangement:** Work From Office is mandatory. - **Shift flexibility:** willingness to work from office and in rotational shifts