Lead Software Engineer - Data Engineer

Be an integral part of an agile team that's constantly pushing the envelope to enhance, build, and deliver top-notch technology products. As a Lead Software Engineer at JPMorganChase within the Commercial & Investment Bank (CIB) – Regulatory Reporting Team, you are an integral part of an agile team that works to enhance, build, and deliver trusted market-leading technology products in a secure, stable, and scalable way. As a core technical contributor, you are responsible for conducting critical technology solutions across multiple technical areas within various business functions in support of the firm’s business objectives. Job responsibilities - Executes creative software solutions, design, development, and technical troubleshooting with ability to think beyond routine or conventional approaches to build solutions or break down technical problems, with a focus on data engineering and Spark-based ETL/ELT - Develops secure high-quality production code in Python/PySpark and Spark SQL, and reviews and debugs code written by others (Spark jobs, SQL logic, and data issues end-to-end) - Drives team adoption of enterprise-authorized AI-assisted engineering practices within the work environment to improve code quality, delivery speed, and operational outcomes (e.g., AI-assisted code review/refactoring, test strategy acceleration, incident/root-cause analysis support), while establishing consistent validation standards (secure coding, peer review, automated testing) and promoting reuse of effective patterns across the team. - Applies knowledge of tools within the Software Development Life Cycle toolchain, including enterprise-authorized AI-assisted development and automation capabilities, to improve the value realized by automation. - Identifies opportunities to eliminate or automate remediation of recurring issues to improve overall operational stability of software applications and systems, including data pipeline reliability and lakehouse maintenance automation - Leads evaluation sessions with external vendors, startups, and internal teams to drive outcomes-oriented probing of architectural designs, technical credentials, and applicability for use within existing systems and information architecture (e.g., EMR/Databricks, lakehouse/table formats, catalog/governance patterns) - Leads communities of practice across Software Engineering to drive awareness and use of new and leading-edge technologies, especially around Spark performance, Iceberg best practices, and data platform operations - Adds to team culture of diversity, opportunity, inclusion, and respect Required qualifications, capabilities, and skills - Formal training or certification on software engineering concepts and 5+ years applied experience - 5+ years of applied experience building production data engineering and/or software engineering solutions (design, development, testing, operations) - Hands-on practical experience delivering system design, application development, testing, and operational stability for large-scale data pipelines - Advanced in one or more programming language(s), with advanced proficiency in Python and strong hands-on experience with PySpark. - Advanced proficiency in Spark SQL and strong SQL fundamentals (data modeling, query optimization, execution plan analysis) - Demonstrated experience leading effective use of approved AI-assisted software development tools (e.g., for coding, code review, test acceleration, troubleshooting) with the ability to set team expectations for validating AI outputs for correctness, performance, and security - Strong understanding of responsible AI use in engineering workflows, including data sensitivity considerations, secure handling of inputs, outputs, and adherence to resiliency and security expectations; experience coaching engineers on safe, compliant adoption within delivery practice - Experience with AWS data management patterns including S3 and AWS Glue Data Catalog (metadata governance, table schema hygiene, discoverability). Would also consider other cloud based Data platform. - Required platform experience: delivering and operating Spark workloads on EMR and or Databricks (tuning, troubleshooting, monitoring, and cost, performance optimization) - Required lakehouse expertise: production experience with Apache Iceberg, including table design and ongoing operations such as partitioning strategy and file layout optimization, schema evolution and compatibility controls, compaction, small-file mitigation, snapshot retention management and metadata maintenance, safe backfills and rewrites, reprocessing patterns - Proficiency in automation and continuous delivery methods (CI CD, automated testing, and repeatable deployments for data pipelines) Preferred qualifications, capabilities, and skills - Kafka familiarity (topic design, producer/consumer patterns, schema evolution/compatibility, and operational considerations) is a plus - Experience with Delta Lake concepts and trade-offs vs. Iceberg - Experience with Spark Structured Streaming and streaming ETL patterns - Working knowledge of Java (interoperability or leveraging existing JVM-based components) - Experience using AI-assisted engineering tools and workflows (e.g., GitHub Copilot, Claude) including spec-driven development, prompt-assisted refactoring, and code review—following enterprise-safe usage patterns