Senior Data Engineer – PySpark/Databricks/Airflow Full Time NEW

Spencer Rose Ltd

Senior Data Engineer – PySpark/Databricks/Airflow

Location: London
Salary: Up to £140,000 basic
Working Arrangement: 5 days per week in the office

An established specialist insurance business is looking for a Senior Data Engineer to join its growing analytics and data engineering function.

This is an opportunity to take ownership of data engineering projects across a modern analytics platform, building scalable data pipelines and delivering high-quality data products that support actuarial analysis, portfolio reporting and wider business decision-making.

Working closely with the Lead Data Engineer and business stakeholders, you will play a key role in developing the data platform, improving existing processes and translating business requirements into effective technical solutions.

Key Responsibilities

• Design, build and maintain scalable data pipelines and data products.

• Use PySpark and modern data engineering technologies to transform raw data into reliable, analysis-ready datasets.

• Develop and improve data orchestration and transformation processes.

• Work directly with business stakeholders to understand requirements and deliver practical technical solutions.

• Support the ongoing development, maintenance and optimisation of the existing data platform.

• Implement automated data quality checks, monitoring and best practices.

• Identify opportunities to improve performance, scalability and efficiency across the platform.

• Work collaboratively with engineering, architecture and business teams to deliver projects end to end.

Skills & Experience

• Strong commercial experience as a Data Engineer, ideally at Senior level.

• Hands-on experience with PySpark and/or Databricks.

• Experience with Airflow is highly desirable, particularly for data pipeline orchestration.

• Experience with DBT or similar data transformation tools would be beneficial.

• Experience working with cloud-based data platforms, ideally within the Microsoft Azure ecosystem.

• Snowflake experience would be advantageous.

• A good understanding of data modelling, data quality and modern data platform architecture.

• Strong communication skills, with the confidence to engage directly with stakeholders and translate business needs into technical solutions.

• A proactive approach, with the ability to take ownership of projects and deliver high-quality work.

Additional Information

This is a stakeholder-facing role where communication and the ability to understand business requirements are just as important as technical capability.

The position requires five days per week in the London office. This is a firm requirement, so applicants must be comfortable with a fully office-based working arrangement.

The salary is up to £140,000 basic, depending on experience.