Technical necessities abstract:
Have expertise designing and constructing data-intensive distributed systemsHave expertise working throughout the fashionable batch and streaming information ecosystemsAn skilled in Java/Scala or Python and have expertise with SQL and BashAble to personal, set up, and steer staff projectsBe in a position to contribute to undertaking administration and undertaking reportingBe a frontrunner/mentor to junior members of the staff regarding engineering best-practices and code-quality
What you’ll do on this function:
Assist design, construct, and handle DRW’s Unified Information Platform and assist its customers.Work carefully with Merchants and Researchers to find out acceptable information sources and implement processes to onboard and handle new information sources for evaluation to unlock future buying and selling alternatives.Design and develop information options to assist uncover, buy, set up, observe utilization, handle rights, and management high quality of knowledge units to deal with the wants of assorted DRW buying and selling groups and methods.Regularly monitor information ingestion pipelines and information high quality to make sure stability, reliability, and high quality of the info. Contribute to the monitoring and high quality management software program and processes.
What you have to on this function:
7+ years of expertise working with fashionable information applied sciences and/or constructing data-intensive distributed systemsExpert stage abilities in Java/Scala or Python with a confirmed means to output high-quality, maintainable, codeStrong familiarity with SQL and BashExperience leveraging and constructing cloud-native applied sciences for scalable information processing.Prior expertise with each batch and streaming programs and an understanding of the restrictions these paradigms imposeExperience with an array of knowledge processing applied sciences (e.g. Flink, Spark, Polars, Dask, and so on.)Expertise with an array of knowledge storage applied sciences (e.g. S3, RDBMS, NoSQL, Delta/Iceberg, Cassandra, Clickhouse, Kafka, and so on.)Expertise with an array of knowledge codecs and serialization programs (e.g. Arrow, Parquet, Protobuf/gRPC, Avro, Thrift, JSON, and so on.)Expertise managing complicated information ETL Pipelines (e.g. Kubernetes, Argo Workflows, Airflow, Prefect, Dagster, and so on.)Prior expertise coping with schema governance and schema evolutionPrior expertise growing information high quality management processes to detect information gaps or inaccuraciesA want to mentor much less skilled team-members and champion each engineering best-practices and excessive code-quality requirements.Robust technical problem-solving skillsProven means to work in an agile, fast-paced setting, prioritize a number of duties and tasks, and effectively deal with the calls for of a buying and selling setting