Mid-sized technology company

Senior Data Engineer

Apply
Full-timeNew York, NYRemote$120k-140k

Company Description

A mid-sized technology company whose data platform deals heavily with financial transaction data. Data engineering works alongside analytics, data science, product and engineering teams.

Job Description

This is a senior individual contributor role for a data engineer who likes deep technical problems. You will take pipelines from raw ingestion all the way to clean, well-modeled data that analysts and other teams can use on their own.

You will have a lot of autonomy. You will shape how pipelines and data models are designed, built and tuned, and you will be a go-to technical resource for the rest of the team. Much of the data involves financial transactions, so experience with ledgers, reconciliation or fraud data is a real advantage.

The work covers the full lifecycle: orchestration, modeling standards, data quality and monitoring, performance and cost tuning, and governance. You will help make the platform reliable, well documented and built on automation rather than one-off fixes.

You will:

  • Design, build and improve ELT data flows on a cloud platform that pull from both internal systems and outside sources
  • Take ownership of how those data flows are scheduled and run, from start to finish
  • Create and maintain analytical models that let analysts serve themselves, and write down the modeling conventions the team follows
  • Add validation, quality checks and alerting so problems in the data are caught early
  • Make workloads faster and cheaper to run by profiling and improving slow or expensive processing
  • Partner with analytics, data science, product and engineering colleagues to translate their needs into technical designs
  • Take part in design discussions and review teammates' code
  • Bring governance and engineering discipline to the platform, including access control, lineage, data classification, version control, CI/CD and automated testing

Job Requirements:

  • 7+ years in data engineering or software engineering, with a track record of building large, multi-source data flows and fixing hard production problems
  • Deep hands-on Databricks experience in production, including setting up compute and scheduling workloads
  • Strong PySpark skills across batch and streaming, with the ability to tune and debug independently
  • Experience on AWS, Azure or GCP, including the basics of storage and networking
  • Very strong SQL, including analytic functions and diagnosing and improving slow queries
  • Very strong Python for building data flows and transformation logic
  • Experience with dbt or a similar transformations-as-code tool, and with an orchestration tool used in production
  • Hands-on experience operating streaming systems such as Kafka in a live environment
  • Demonstrated ability to design schemas for analytics and reporting
  • Sound engineering habits: Git workflows, testing, code review and CI/CD applied to data
  • Communicates clearly with technical and non-technical audiences
  • Background with payments or fintech data such as transactions, ledgers, reconciliation, fraud or risk (nice to have)
  • Familiarity with data catalog or lineage tools, BI platforms, or infrastructure as code (nice to have)
  • Experience supporting machine learning work, such as feature pipelines or training data infrastructure (nice to have)

Apply for this Position