# Senior Manager, AI Engineer/ Solutions & Engineering — Bristol Myers Squibb

- Location: Princeton - NJ - US
- Posted pay: $154k–$186k/yr
- Posted: Nov 13, 2026
- Status: open (checked Nov 21, 2026)
- Apply (employer's site): https://bristolmyerssquibb.wd5.myworkdayjobs.com/BMS/job/Princeton---NJ---US/Senior-Manager--AI-Engineer--Solutions---Engineering_R1608179-1
- Employer record: https://visajobs.io/company/bristol-myers-squibb.md

## Sponsorship history
- Tier: Employer history only
- With this exact title: 0

## Description
At Bristol Myers Squibb, our employees often ask, “Who are you working for?”—a question that fuels collaboration, accountability, and urgency in our work. Our purpose-driven culture inspires us to discover, develop, and deliver innovative medicines to prevail over serious diseases. We offer uniquely interesting and meaningful work, opportunities for growth, and a supportive environment that values inclusion, wellbeing, flexibility, and comprehensive benefits. This is work that transforms the lives of patients, and the careers of those who do it.
At BMS, digital innovation and technology are central to our vision of transforming patients’ lives through science. We are applying AI to accelerate content creation, speed up decision making, and enable more personalized customer engagement, increasingly through enterprise AI assistants and agents that commercial teams use in their daily work.
We are looking for a hands-on AI Engineer who solves real business problems with LLMs. You will design how models are prompted, grounded, equipped with tools, evaluated, and governed, and you will build the solutions that run on top of them, using coding agents to move fast. You will understand how models and agent harnesses work well enough to know when a solution will succeed, when it will fail, and how to design around both.
Key Responsibilities
Build and ship LLM-powered solutions (assistant extensions, connectors, agentic workflows, and custom applications) from prototype to production
Build evaluations first: define what “good” means for each use case, create golden datasets, graders, and regression suites, and use them to drive every prompt, tool, and model change
Build with coding agents: use agents such as Claude Code as a daily development tool, writing specs, directing agents, and reviewing and testing their output, while owning correctness and quality
Design around the harness: understand how AI assistant platforms run the agent loop (tool calls, context assembly, permissions, sandboxing) and shape solutions to fit
Engineer context deliberately: system prompts, reusable instructions, memory, retrieval, structured outputs, and long-context strategy
Choose the right build pattern: platform extension vs. connector vs. custom app vs. API-based agent, and when not to use an agent
Select the right model for each task by weighing capability, cost, latency, and risk, using your own evals as evidence
Implement guardrails and human-in-the-loop controls suited to regulated pharma content (MLR review, compliance, auditability, data boundaries)
Monitor and improve quality, cost, latency, and adoption of deployed solutions using production traces and feedback
Translate business needs into technical solutions with marketing, commercial, and IT partners
Enable the business: explain what AI can do, where it breaks, and what it is worth
Raise the team’s practice: share reusable patterns, review peers’ work, and track new models and platform capabilities
Required Knowledge and Skills
Evaluation and quality engineering
Designing evals for non-deterministic systems: golden datasets, rubric and LLM-as-judge graders, human review, and regression testing
Evaluating each layer separately: retrieval, tool use, multi-step agent trajectories, and final output
Validating the judges themselves: calibrating LLM graders against human labels and spotting grader bias
Running evals in CI so that changes to prompts, tools, skills, or model versions are measured before release
Turning evaluation results into improvements (error analysis, failure categorization, targeted fixes)
Working with coding agents
Current, hands-on daily use of coding agents such as Claude Code (or Codex, Cursor, or similar) to design, build, test, and refactor real software
Spec-driven workflows: writing clear specs and acceptance tests, breaking work into agent-sized tasks, and running agents in parallel or over long tasks
Configuring agents: project instructions, skills, MCP servers, hooks, and permissions
Reviewing agent output critically. You can explain and defend every line you ship, and you know when an agent is going off track.
Applied LLM engineering
Proven LLM solutions in production, not just prototypes
Strong Python (ideally TypeScript too), API-driven development, Git, CI/CD, and a cloud platform (AWS, Azure, or GCP)
Advanced prompt and context engineering: system prompt design, few-shot patterns, structured outputs, reasoning and long-context strategies
Retrieval and grounding: chunking, hybrid search, reranking, metadata filtering, and knowing when connectors or direct file access replace a vector DB
Agentic patterns: tool and function calling, multi-step workflows, MCP, and judgment on when not to use an agent
Working with unstructured and semi-structured data (documents, PDFs, content assets)
Understanding how models and harnesses work
Transformer fundamentals: tokenization, attention, context windows, and sampling
The training stack (pretraining, instruction tuning, RLHF/RLAIF) and how reasoning models differ
How an agent harness works: the loop, tool exposure, context management, and permissions
Failure modes (hallucination, sycophancy, position bias, prompt injection, tool misuse) and the mitigation for each
Reasoning about cost, latency, and quality from first principles
Engineering practices
LLMOps: tracing, observability, versioning of prompts and models, and cost and usage management
Security basics: prompt injection via tools and documents, data leakage, PII, least-privilege access
Desirable Experience
Enterprise AI assistant platforms: hands-on experience building on Claude Desktop (Chat, Cowork, Code; skills, projects, plugins, connectors), ChatGPT Enterprise (custom GPTs, connectors, projects), Gemini Enterprise / Gemini for Workspace, or similar harnesses
Building or configuring MCP servers and integrating enterprise systems as connectors
Agent SDK or API-based agent development for cases that outgrow the assistant platform
Pha

> History, not a promise: past H-1B filings do not mean an employer will sponsor you or a given role. Data: U.S. DOL OFLC LCA disclosure (FY2024–FY2026 for role counts, certified LCAs) and the USCIS H-1B Employer Data Hub. Not legal advice. Methodology: https://visajobs.io/methodology
