Virtusa

Lead Software Engineer

  • New York
  • temporary
  • Posted 1 month ago
  • Staffing / consulting
Apply on Virtusa's site ↗Checked today on the employer's career site. We never take applications.

Data Quality Automation Analyst (1) (Databricks-Centric)

Role Overview

Design and automate data validation within Databricks pipelines to ensure accurate migration and ongoing data reliability.

Key Responsibilities

Build automated data quality checks in PySpark

Develop reconciliation framework between:

Legacy SQL outputs

New Databricks outputs

Implement rule-based validation for healthcare datasets

Automate profiling and anomaly detection

Integrate quality checks into pipeline workflows

Build DQ scorecards and dashboards

Support cutover validation

Required Skills

Strong SQL & PySpark

Experience with Databricks validation frameworks

Experience implementing automated DQ checks

Strong reconciliation expertise

Healthcare data validation experience preferred

Similar open jobs

All H-1B software engineer jobs in New York →