Data Engineer

About this role

About the role: - Build and maintain data pipelines that handle large-scale datasets that exceed available memory - Make architectural decisions between database solutions and script-based approaches based on use case requirements - Implement streaming data processing systems with proper checkpointing and fault tolerance - Design data workflows that incorporate chunking, parallelism, and recovery from failures - Establish quality controls and deduplication processes to ensure dataset integrity - Evaluate how data engineering choices impact model training, validation, and performance What you'll work on: - Processing and transforming data in formats like Parquet and WebDataset - Integrating with object storage systems including S3, GCS, and R2 - Using Hugging Face Datasets and related tools for efficient data handling - Building systems that catch and handle corrupted or malformed records gracefully - Creating deduplication logic that maintains dataset uniqueness and prevents evaluation contamination What we're looking for: - Proficiency in Python with a track record of working on datasets larger than RAM - Solid grasp of data engineering principles including streaming, chunking, parallelism, and checkpointing - Understanding of how dataset decisions cascade into model behavior and evaluation outcomes - Focus on data quality as a core responsibility Desirable experience: - Background curating training data for large language models, speech systems, or computer vision applications - Familiarity with embeddings, vector search tools such as FAISS or Milvus, and semantic deduplication techniques - Hands-on experience with audio and video processing using ffmpeg, torchaudio, or forced alignment tools - Work generating synthetic data via LLMs or 3D asset creation with Blender or Unreal Engine - Native or near-native proficiency in one or more Indian languages