Data Quality Engineer (Databricks)

Develop Delta Lake metric aggregation structures supporting enterprise quality reporting.

• Calculate and publish:
  • Data Quality Index (DQI) scores.
  • Dimension-level quality metrics.
  • Rule pass/fail rates.
  • Dataset compliance scores.
  • SLA adherence indicators.
• Provide curated outputs to support Power BI quality dashboards and executive reporting.

Monitoring, Alerting and Predictive Quality Management:

• Configure automated quality monitoring and alerting mechanisms.
• Implement threshold-based notifications using:
  • Databricks SQL Alerts.
  • Azure Monitor integrations.
• Develop predictive risk scoring models to identify datasets at risk of future quality degradation.
• Support proactive quality management and operational intervention activities.

Root Cause Analysis and Remediation Support:

Apply Databricks machine learning and pattern analysis techniques to profiling and rule execution outputs.

• Support AI-assisted root cause analysis across established remediation categories.
• Identify recurring quality issues, systemic defects, and process breakdowns.
• Produce prioritised remediation datasets for business and operational stakeholders.
• Export remediation outputs to Power BI and Excel to support:
  • Remediation Tiering Matrix.
  • Prioritisation Scoring Models.
  • Governance reporting processes.

Governance, Compliance and Stakeholder Engagement:

• Collaborate with Data Modellers and Data Catalogue Specialists to ensure quality controls align with authoritative data definitions and metadata standards.
• Support DDA and DGE governance processes by producing required quality artefacts and evidence.
• Maintain documentation, version control, and audit trails for all quality assets, rules, models, and processes.
• Participate in quality reviews, governance forums, and stakeholder workshops.

Required Skills and Experience:

• Strong experience designing and implementing enterprise Data Quality frameworks.
• Advanced Databricks engineering experience.
• Strong PySpark development skills.
• Experience with:
  • Delta Lake.
  • Unity Catalog.
  • Databricks Workflows and Jobs.
  • Databricks SQL.
• Experience building scalable data validation and quality rule frameworks.
• Knowledge of machine learning techniques for anomaly detection and data quality monitoring.
• Experience using MLflow for model management and deployment.
• Strong understanding of data governance, metadata management, and data lifecycle processes.
• Experience integrating data quality metrics into reporting platforms such as Power BI.
• Knowledge of cloud-based data engineering and modern Lakehouse architectures.

Deliverables:

• Configured Databricks Data Quality environment.
• Enterprise Data Quality Rule Factory.
• AI-assisted profiling notebooks and baseline assessment outputs.
• Automated data quality validation and gating processes.
• Data cleansing and remediation pipelines.
• Failed Record Register and reprocessing workflows.
• Data Quality metric aggregation tables.
• DQI reporting feeds and dashboard datasets.
• Predictive quality monitoring and alerting solutions.
• Root Cause Analysis and remediation support datasets.
• Governance, audit, and compliance artefacts supporting DDA and DGE reviews.

Salt is acting as an Employment Business in relation to this vacancy.