Working with colleagues to understand and implement requirements
Securing the data supply chain, understanding how data is ingested from different sources and combined / transformed into a single data set.
Understanding how to analyse, cleanse, join and transform data.
Implementing designed / specified solutions into the chosen platform
Working with colleagues to ensure that the onprem/cloud infrastructure available is capable of meeting the solution requirements.
Planning, designing and conducting tests of the implementations, correcting errors and retesting to achieve an acceptable result.
Appreciate how to manage the data including; security, archiving, structure and storage.
Key Experience and Qualifications
Degree level education in Mathematics, Scientific, Computing or Engineering discipline or equivalent experience
6+ years of experience at various levels of Data Engineering roles including 3 years in technical lead role managing end to end solution.
Experience in designing solutions using databases and data storage technology using RDBMS (MS SQL Server)
Experience building and optimizing Big Data data pipelines, architectures and data sets using MS Azure data management and processing components through IaaS/PaaS/SaaS implementation models implemented through custom
Proficiency in Python scripting. Experienced in using different python modules used for data munging
Be up to date with data processing technology / platforms such as Spark (Databricks)
Experience in Data Modeling for optimizing the solution performance
Experienced in Azure DevOps
Experience with ETL and/or data integration tool MuleSoft
Good understanding of infrastructure components and their fit in different types of data solutions
Experience of designing solutions deployed on Microsoft and Linux operating systems
Experience of working in an agile environment, within a selforganising team.