Data Infrastructure Engineer (Rust)
Job Description
High-performance data infrastructure is the backbone of cutting-edge AI, and Rust is at its forefront. As a Data Infrastructure Engineer specializing in Rust, you'll be pivotal in teaching AI models to build, optimize, and secure the next generation of data pipelines and distributed systems, leveraging Rust's unparalleled safety and speed.
Why This Role Matters
Your deep expertise in Rust for data infrastructure will directly influence the AI's ability to generate robust, concurrent, and memory-safe code for critical data workloads. By providing high-quality training data, you'll empower AI to master complex Rust paradigms for distributed computing, pushing the boundaries of what's possible in data processing.
Key Responsibilities
Review and evaluate Rust code for distributed data processing frameworks (e.g., Apache Arrow, DataFusion, Tokio-based systems).
Create detailed training examples demonstrating efficient memory management and concurrency patterns in Rust for large-scale data.
Evaluate AI-generated Rust code for correctness, performance bottlenecks, and adherence to idiomatic Rust practices in data pipelines.
Develop scenarios and solutions for common challenges in distributed systems, such as fault tolerance, consistency, and message passing using Rust.
Provide expert feedback on Rust implementations of data serialization/deserialization, network protocols, and storage interfaces.
Design and validate Rust code snippets for high-performance computing (HPC) infrastructure, focusing on zero-cost abstractions and unsafe Rust when appropriate.
Ideal Qualifications
Expert-level proficiency in Rust, including async/await, lifetimes, traits, and the standard library.
Extensive experience designing and implementing distributed systems or high-performance data pipelines in Rust.
• Familiarity with Rust's ecosystem for data science and distributed computing (e.g., \`rayon\