Lead Software Engineer - Data Engineer
Apply on JP Morgan Chase's site
Description -----------
Be an integral part of an agile team that's constantly pushing the envelope to enhance, build, and deliver top-notch technology products.
As a Lead Software Engineer at JPMorganChase within the Commercial & Investment Bank (CIB) – Regulatory Reporting Team, you are an integral part of an agile team that works to enhance, build, and deliver trusted market-leading technology products in a secure, stable, and scalable way. As a core technical contributor, you are responsible for conducting critical technology solutions across multiple technical areas within various business functions in support of the firm’s business objectives.
Job responsibilities
- Executes creative software solutions, design, development, and technical troubleshooting with ability to think beyond routine or conventional approaches to build solutions or break down technical problems, with a focus on data engineering and Spark-based ETL/ELT
- Develops secure high-quality production code in Python/PySpark and Spark SQL, and reviews and debugs code written by others (Spark jobs, SQL logic, and data issues end-to-end)
- Drives team adoption of enterprise-authorized AI-assisted engineering practices within the work environment to improve code quality, delivery speed, and operational outcomes (e.g., AI-assisted code review/refactoring, test strategy acceleration, incident/root-cause analysis support), while establishing consistent validation standards (secure coding, peer review, automated testing) and promoting reuse of effective patterns across the team.
- Applies knowledge of tools within the Software Development Life Cycle toolchain, including enterprise-authorized AI-assisted development and automation capabilities, to improve the value realized by automation.
- Identifies opportunities to eliminate or automate remediation of recurring issues to improve overall operational stability of software applications and systems, including data pipeline reliability and lakehouse maintenance automation
- Leads evaluation sessions with external vendors, startups, and internal teams to drive outcomes-oriented probing of architectural designs, technical credentials, and applicability for use within existing systems and information architecture (e.g., EMR/Databricks, lakehouse/table formats, catalog/governance patterns)
- Leads communities of practice across Software Engineering to drive awareness and use of new and leading-edge technologies, especially around Spark performance, Iceberg best practices, and data platform operations
- Adds to team culture of diversity, opportunity, inclusion, and respect
Required qualifications, capabilities, and skills
- Formal training or certification on software engineering concepts and 5+ years applied experience
- 5+ years of applied experience building production data engineering and/or software engineering solutions (design, development, testing, operations)
- Hands-on practical experience delivering system design, application development, testing, and operational stability for large-scale data pipelines
- Advanced in one or more programming language(s), with advanced proficiency in Python and strong hands-on experience with PySpark.
- Advanced proficiency in Spark SQL and strong SQL fundamentals (data modeling, query optimization, execution plan analysis)
- Demonstrated experience leading effective use of approved AI-assisted software development tools (e.g., for coding, code review, test acceleration, troubleshooting) with the ability to set team expectations for validating AI outputs for correctness, performance, and security
- Strong understanding of responsible AI use in engineering workflows, including data sensitivity considerations, secure handling of inputs, outputs, and adherence to resiliency and security expectations; experience coaching engineers on safe, compliant adoption within delivery practice
- Experience with AWS data management patterns including S3 and AWS Glue Data Catalog (metadata governance, table schema hygiene, discoverability). Would also consider other cloud based Data platform.
- Required platform experience: delivering and operating Spark workloads on EMR and or Databricks (tuning, troubleshooting, monitoring, and cost, performance optimization)
- Required lakehouse expertise: production experience with Apache Iceberg, including table design and ongoing operations such as partitioning strategy and file layout optimization, schema evolution and compatibility controls, compaction, small-file mitigation, snapshot retention management and metadata maintenance, safe backfills and rewrites, reprocessing patterns
- Proficiency in automation and continuous delivery methods (CI CD, automated testing, and repeatable deployments for data pipelines)
Preferred qualifications, capabilities, and skills
- Kafka familiarity (topic design, producer/consumer patterns, schema evolution/compatibility, and operational considerations) is a plus
- Experience with Delta Lake concepts and trade-offs vs. Iceberg
- Experience with Spark Structured Streaming and streaming ETL patterns
- Working knowledge of Java (interoperability or leveraging existing JVM-based components)
- Experience using AI-assisted engineering tools and workflows (e.g., GitHub Copilot, Claude) including spec-driven development, prompt-assisted refactoring, and code review—following enterprise-safe usage patterns
About JP Morgan Chase
JP Morgan Chase is a global financial services provider that offers investment banking, asset management, treasury, and other services.
More jobs at JP Morgan Chase
- Product Delivery Manager- Asset Based Lending Wholesale Lending Services
- Lead Software Engineer — Workflow Orchestration (React | Camunda | Java/Python) Full Stack
- Lead Software Engineer (SWES04)- SDET with Playwright
- Vice President, Strategic Program Manager - Affluent Program Office
- Lead Software Engineer- Data Engineer/Pyspark/Databricks
- Full-Stack Java/Python React Software Engineer III - Trading Platform
- Sr. Lead Software Engineer: Data Engineering
- Software Engineer III - Automation Engineer
- Site Reliability Engineer III
- Sr Director of Security Engineering