Company Description
A mid-sized technology company whose data platform deals heavily with financial transaction data. Data engineering works alongside analytics, data science, product and engineering teams.
Job Description
This is a senior individual contributor role for a data engineer who likes deep technical problems. You will take pipelines from raw ingestion all the way to clean, well-modeled data that analysts and other teams can use on their own.
You will have a lot of autonomy. You will shape how pipelines and data models are designed, built and tuned, and you will be a go-to technical resource for the rest of the team. Much of the data involves financial transactions, so experience with ledgers, reconciliation or fraud data is a real advantage.
The work covers the full lifecycle: orchestration, modeling standards, data quality and monitoring, performance and cost tuning, and governance. You will help make the platform reliable, well documented and built on automation rather than one-off fixes.
You will:
- Design, build and improve ELT data flows on a cloud platform that pull from both internal systems and outside sources
- Take ownership of how those data flows are scheduled and run, from start to finish
- Create and maintain analytical models that let analysts serve themselves, and write down the modeling conventions the team follows
- Add validation, quality checks and alerting so problems in the data are caught early
- Make workloads faster and cheaper to run by profiling and improving slow or expensive processing
- Partner with analytics, data science, product and engineering colleagues to translate their needs into technical designs
- Take part in design discussions and review teammates' code
- Bring governance and engineering discipline to the platform, including access control, lineage, data classification, version control, CI/CD and automated testing
Job Requirements:
- 7+ years in data engineering or software engineering, with a track record of building large, multi-source data flows and fixing hard production problems
- Deep hands-on Databricks experience in production, including setting up compute and scheduling workloads
- Strong PySpark skills across batch and streaming, with the ability to tune and debug independently
- Experience on AWS, Azure or GCP, including the basics of storage and networking
- Very strong SQL, including analytic functions and diagnosing and improving slow queries
- Very strong Python for building data flows and transformation logic
- Experience with dbt or a similar transformations-as-code tool, and with an orchestration tool used in production
- Hands-on experience operating streaming systems such as Kafka in a live environment
- Demonstrated ability to design schemas for analytics and reporting
- Sound engineering habits: Git workflows, testing, code review and CI/CD applied to data
- Communicates clearly with technical and non-technical audiences
- Background with payments or fintech data such as transactions, ledgers, reconciliation, fraud or risk (nice to have)
- Familiarity with data catalog or lineage tools, BI platforms, or infrastructure as code (nice to have)
- Experience supporting machine learning work, such as feature pipelines or training data infrastructure (nice to have)