Apache Spark Developer

LocationMelbourne, Australia
HoursFlexible / negotiable (work from home allowed)
RemunerationAttractive
ConditionContract with a view to permanent

Overview

Qabsu is looking for an Apache Spark developer who can lead, be depended upon and knows how to fully exploit the potential of Spark for our customers. This will entail designing Spark applications that are of exceptional production quality, and can run on environments such as EMR, Glue and DataBricks.

This role does require Australian based candidates with full working rights and will be subject to official background checks.

Responsibilities

  • Create PySpark jobs for data transformation and aggregation
  • Produce unit tests for Spark transformations and helper methods
  • Write docstring-style documentation with all code
  • Design, implement and work with existing data processing pipelines
  • Maintain good lint, coverage and cyclomatic complexity ratings of code
  • Instill and maintain good hygiene of git repositories

Requirements

  • Python (with a focus on the functional programming paradigm)
  • pytest, pylint, radon, VSCode, PyCharm, boto, make, cli
  • Exceptional knowledge of Apache Spark 3.x
  • Ability to demonstrate working knowledge with Kafka and related technologies
  • Ability to demonstrate first principles of Spark
  • Spark and Spark query tuning and performance optimisation
  • Experience working with AWS, S3, DynamoDB, EMR, DataBricks, Glue
  • Deep understanding of distributed systems (e.g. CAP theorem, partitioning, replication, consistency, and consensus)
  • Proficiency in English
  • Communication and negotiation skills
  • Ability to build rapport
  • Time management and planning skills
  • 3-5 years+ of previous experience using Spark with provable qualifications

About Qabsu

Qabsu is an exciting and energetic IT Services and Consulting company incorporated in Australia, with the view of providing exceptional value to our customer base. With a collective experience of 70 years+ in fields ranging from retail to public sector, we pride ourselves on looking after our staff and ensuring the capability our consultants bring to each engagement is far above and beyond exceptional.