Distributed Data and Spark Evaluation Specialist
Job Description
IXO is engaging specialists to evaluate distributed data pipelines that remain reliable at scale. Your contribution is technical work: make the reasoning inspectable, identify substantive errors and provide evidence that supports a reliable assessment.
Work you will do
• Develop or assess ingestion, transformation, storage and processing designs across relevant big-data systems.
• Check partitioning, joins, distributed execution and data quality, investigating pipeline failures and performance limits.
• Document architecture and verification evidence, including Spark-specific internals, table formats or cluster behavior in the specialist track.
• Evaluate Spark SQL/DataFrames/Structured Streaming, partitioning, broadcast joins, skew, shuffles, serialization, cluster sizing and table-format behavior.
Qualification routes
Qualify through the assessment route below or the additional specialist route. Each route has its own background and regional criteria; these routes are alternatives.
Assessment route requirements
• Practical large-scale pipeline engineering, advanced Python and knowledge of both relational and NoSQL data management and optimization.
• Use distributed processing such as Flink, Hadoop or Spark and apply ETL, warehousing and data-modelling principles.
Preferred background
• Cloud data platforms on Azure, AWS or GCP, global/startup environments, and understanding of data-science or ML-operations workflows.
Additional specialist route
The Spark specialist route favors six or more big-data years including four in production Spark, Scala and PySpark, Catalyst/Tungsten/AQE and at least one of Hudi, Iceberg or Delta. Databricks/EMR/Dataproc/Synapse are relevant; Kafka, Airflow or dbt are useful.
Deliverables
Submit the completed technical artifact or assessment with its supporting evidence, explicit assumptions, reproducible checks where applicable, and concise reasons for each material judgment. Address review findings within the agreed scope.
Location and schedule
Additional specialist route eligibility: USA, UK, Canada, Germany, Netherlands. Remote assignments are scheduled by agreement, with no guaranteed weekly volume. IXO confirms the applicable timing before work.
Pay and working terms
$85 • $135/hr USD. The agreed hourly rate, scope, schedule and acceptance criteria are confirmed before work starts. Applying does not guarantee an assignment. Use public, licensed or otherwise authorized material only; do not submit confidential employer information, personal data or restricted research.