IAG Defence and National Security (DNS)
Lead Data Engineer
🇦🇺Canberra, Australian Capital Territory, Australia23 hours ago
SeniorFull TimeEN
IAG Defence & National Security (DNS) is part of IAG, a specialist advisory firm supporting complex infrastructure, Defence and national security programs across Australia.
Our DNS team works closely with Commonwealth agencies and industry partners to deliver specialist expertise into nationally significant programs. Our success depends on our ability to identify, engage and mobilise highly specialised talent quickly and effectively. IAG is recognised as Veteran Friendly Employer under the Veteran Employment Program.
The Australian Signals Directorate (ASD) is seeking an EL1 Equivalent Level Data Engineer (Data Platform Engineer). The Platform Engineer enables efficient delivery by aligning development practices to business priorities, minimising operational complexity and lifecycle management overhead across applications and supporting infrastructure. The Data Platform Engineer owns end-to-end data platform outcomes, delivering scalable, resilient, and observable ingestion, streaming, storage, and analytics capabilities that underpin mission-critical intelligence and cyber security operations.
The successful candidate will be able to:
Contribute to the optimisation and maintenance of data ingestion pipelines including both stream and batch technologies. Develop and manage data storage table and message schemas for various event formats. Support Hadoop and similar technologies to ensure data is stored and accessed efficiently. Ensure the effective implementation of role based access controls across data lake components through technical enforcement and audit. Identify, report and remediate platform issues through proactive monitoring of health and performance metrics. Identify process improvements, and contribute to data ingestion standards and definition of best practice. Communicate and report data ingestion and analysis related risk and issues to the project manager. Enhance and configure highly scalable, resilient, and observable data analysis platforms Observable data ingestion pipelines including serialisation, transformation, enrichment and store Manage multiple, concurrent tasks with competing demand Motivate and mentor teams, aligning complementary skills and allocate resources in a manner that delivers results. Installation, optimisation, configuration, and operational management of search and analytical clusters
Criteria considered essential for this role is as follows:
Hold or obtain a TSPV AGSVA Clearance
Demonstrated expertise in the installation, configuration, optimisation, and operational management of large-scale search and analytical clusters, ensuring performance, resilience, security, and availability across their full operational lifecycle. Accountable experience delivering resilient, scalable data streaming capabilities using Kafka, NiFi, and Spark to enable real-time analytics and mission-critical data flows. Demonstrated competence in software development using Python, Scala, and Java, applying sound engineering practices to build, integrate, and maintain production-grade systems. Demonstrated proficiency with infrastructure automation frameworks, including Ansible, Puppet, and Chef, alongside the design and operation of CI/CD pipelines to enable reliable, repeatable, and scalable delivery. Demonstrated strong hands-on capability in Linux system administration, including performance optimisation, fault diagnosis, and troubleshooting across production environments. Demonstrated experience designing, implementing, and supporting data pipelines across their full lifecycle, ensuring reliability, performance, and operational stability. What we offer
With offices in Perth, Brisbane, Melbourne, Sydney, and Canberra, we’re a tight-knit team, working across some of Australia’s most critical infrastructure and Defence projects. We come from diverse backgrounds and pride ourselves on our flexible, supportive and collaborative team culture. We can offer great work, great people, great culture, great conditions, and an all-round great time, whilst you go about developing a great career in this space.
As part of IAG, you’ll be joining a team of more than 250 professionals’ working across a range of integrated workstreams in a friendly and approachable environment.
