Consulting/Principal Software Engineer

We are seeking a dedicated and skilled developer to join our team of Analytics Engineers and Data Analysts working with various stakeholders.  

As a Senior Azure Analytics Engineer, you will be responsible to provide for designing, implementing, and maintaining sophisticated software solutions, leveraging your expertise in Python, PySpark, SQL, and API integration on Azure platform services like ADF and Databricks. 

You will also leverage various AI Agents like Azure Open AI, Databricks Assistant, Genie and Copilot to improve productivity. 

About the Team: 

Team is tech hub for rest of the organization and hence working across multiple domains – US Healthcare, Insurance, Government etc.  

Team primarily works on data extraction, manipulation, loading, and analytics using technologies like Python, PySpark, Bigdata – HPCC, language ECL, Talend, Azure tools (Synapse, ADF etc), Power BI, Essbase etc. 

Key Responsibilities: 

• Design and Develop Scalable Data Pipelines: Architect and implement robust, scalable, and high-performance data pipelines using Azure Data Factory (ADF) and Azure Databricks to ingest, transform, and load data from various sources. 

• Data Integration and Transformation: Build complex data workflows and transformation logic using PySpark and SQL, ensuring efficient processing of structured and unstructured data. 

• Data Modeling and Architecture: Design and maintain data models (star/snowflake schemas), data lakes, and data warehouses to support analytics, reporting, and machine learning use cases. 

• Performance Optimization: Monitor and optimize data pipeline performance, troubleshoot bottlenecks, and implement best practices for data partitioning, caching, and parallel processing. 

• Data Quality and Governance: Implement data validation, cleansing, and quality checks. Ensure compliance with data governance policies and maintain data lineage and metadata. 

• Collaboration and Stakeholder Engagement: Work closely with data scientists, analysts, product managers, and business stakeholders to gather requirements, deliver solutions, and support data-driven decision-making. 

• Documentation and Knowledge Sharing: Create and maintain comprehensive documentation for data pipelines, architecture, and processes. Mentor junior engineers and contribute to team knowledge sharing. 

• Innovation and Continuous Improvement: Stay current with emerging technologies in data engineering and cloud computing. Evaluate and recommend new tools and frameworks to improve efficiency and scalability. 

• AI Integration: Working knowledge of AI‑assisted tools like Databricks Assistant, Genie, Copilot, etc. (e.g., for data analysis, code generation, automation, and productivity) and the ability to effectively leverage them to improve solution delivery, efficiency, and client outcomes.  

Tech Stack: 

Must haves: Azure Data Factory, Azure Databricks, Python, PySpark and SQL. 

Good to have: Machine Learning Models. 

Requirements:  

• Should have 10+ years of experience. 6+ years of relevant experience. 

• Knowledge of SDLC process like requirement gathering, design, implementation (Coding), testing, deployment, and maintenance. Solid understanding of software development principles, design patterns, and best practices. 

• Proficient in requirement gathering and documentation. 

• Excellent communication skills – verbal and written. 

• Strong problem-solving skills. 

• Proven experience as a Senior Analytics Engineer, working on complex projects and delivering high-quality software solutions. 

• Strong proficiency in Python and PySpark programming language and its libraries 

• In-depth knowledge and experience in SQL, including designing and optimizing complex queries, and working with relational databases such as MySQL, PostgreSQL, or Oracle. 

• Proficiency in working with version control systems like Git. 

• Proficiency in Unity Catalog and various AI Agents like Copilot, Databricks Genie, etc. 

• Strong problem-solving and analytical skills, with the ability to quickly identify and resolve issues. 

• Excellent communication and collaboration skills, with the ability to work effectively in a team environment. 

• Experience with Agile development methodologies.  

• Must have a positive attitude and desire to learn new processes and technologies

We know your well-being and happiness are key to a long and successful career. We are delighted to offer country specific benefits. Click here to access benefits specific to your location.

We are committed to providing a fair and accessible hiring process. If you have a disability or other need that requires accommodation or adjustment, please let us know by completing our Applicant Request Support Form or please contact 1-855-833-5120.

Criminals may pose as recruiters asking for money or personal information. We never request money or banking details from job applicants. Learn more about spotting and avoiding scams here.

Please read our Candidate Privacy Policy.

We are an equal opportunity employer: qualified applicants are considered for and treated during employment without regard to race, color, creed, religion, sex, national origin, citizenship status, disability status, protected veteran status, age, marital status, sexual orientation, gender identity, genetic information, or any other characteristic protected by law.

USA Job Seekers:

EEO Know Your Rights.