Mission Overview:
Keystone Solutions is seeking a Senior Data Engineer for a consultancy mission at a client site. You will be part of a dynamic team working on the Data Exchange Platform (DEP) project, which involves building an on-premise data platform for a governmental organization. This role emphasizes the importance of security, governance, and data sovereignty.
Main Responsibilities:
-
Design and deploy data ingestion pipelines from heterogeneous sources: files, APIs, relational databases.
-
Implement pipelines for incremental and streaming ingestion.
-
Ensure reliability, idempotence, and traceability of each pipeline.
-
Develop transformation models on Bronze, Silver, and Gold layers (Medallion architecture).
-
Implement best practices for data historization (e.g., SCD2, upsert, append only).
-
Develop exports to Delta Lake.
-
Manage and maintain Delta Lake tables.
-
Guarantee transactional consistency of data in a multi-domain environment.
-
Implement and maintain orchestration assets.
-
Monitor runs, manage errors, and automate alerts.
-
Push technical and business metadata to the data platform.
-
Maintain end-to-end lineage.
-
Contribute to the drafting of DUA, SLA, and associated governance documents.
-
Develop endpoints to expose the Gold layer to BI dashboards.
-
Collaborate with reporting teams for BI dashboard feeding.
Required Technical Skills:
-
Proficiency in Python.
-
Proficiency in SQL.
-
Proven experience with dbt-core (models, snapshots, macros, tests, profiles).
-
Good knowledge of DuckDB as an embedded analytical query engine.
-
Experience with Delta Lake (ACID transactions, time travel, OPTIMIZE/VACUUM).
-
Knowledge of relational databases: MSSQL, PostgreSQL (advanced SQL, CDC).
-
Experience with a data orchestrator: Dagster, Airflow, or equivalent.
-
Comfortable with Linux/OpenShift and using PowerShell (Windows dev environment).
-
Familiarity with governance tools: DataHub, OpenMetadata, or equivalent.
-
Experience in BI development.
Preferred Skills:
-
Experience with Kafka/Debezium for real-time data capture.
-
Knowledge of Spark (Spark SQL, Thrift Server, Beeline).
-
Experience with Power BI Report Server (PBIRS) or Apache Superset.
-
Awareness of data security in restricted access environments.
-
Knowledge of Lakehouse concepts (Medallion architecture), DWH, and Datalake.
Soft Skills:
-
Rigorous and autonomous in managing complex multi-domain pipelines.
-
Ability to document technical decisions and simplify for non-technical stakeholders.
-
Team spirit.
-
Curious and solution-oriented mindset.
What We Offer:
-
A stable mission that is long-term.
-
A high-impact project in a secure and technically stimulating environment.
-
A modern and open-source stack: DLT, dbt, DuckDB, Delta Lake, Dagster.
-
An experienced and supportive team focused on quality and best practices.
-
Opportunities for skill enhancement in governance, orchestration, and lakehouse.
If you are ready to tackle technical and strategic challenges in a dynamic consultancy environment, apply today .
Duration: As soon as possible - 31/12/2026 5 months • (full time)
Skills required:
-
Dagster - Level: Junior - Most recent: Any time
-
Data acquisition (ETL, ELT, ...) - Level: Confirmed - Most recent: Any time
-
Data architecture - Level: Junior - Most recent: Any time
-
data engineering - Level: Confirmed - Most recent: Any time
-
Data Governance - Level: Confirmed - Most recent: Any time
-
DataHub - Level: Junior - Most recent: Any time
-
datalake - Level: Junior - Most recent: Any time
-
dbt - Level: Junior - Most recent: Any time
-
Kafka - Level: Junior - Most recent: Any time
-
Metadata management - Level: Confirmed - Most recent: Any time
-
POWER BI - Level: Junior - Most recent: Any time
-
Python (from a Data Engineer perspective), Pandas & Apache Spark - Level: Confirmed - Most recent: Any time
-
SPARK - Level: Junior - Most recent: Any time
-
SQL - Level: Confirmed - Most recent: Any time
Language requirements:
Dutch
Level Active knowledge
English
Level Active knowledge
French
Level Native
