Germany

Master’s Degree

Bachelor of Science (BS)

Madiha Khalid

Senior Data Engineer

Skills: Unstructured Data · Passionate about Work · Datadog · Cloud Infrastructure · Data Ingestion · Communication · Defining Requirements · Software as a Service (SaaS) · AWS CloudFormation · Transact-SQL (T-SQL) · Extract, Transform, Load (ETL) · Apache Spark · Apache Kafka · Amazon Web Services (AWS) · Python (Programming Language)

- Design, developed and maintain ETL Pipeline using Python, Docker and Jenkins over AWS Infrastructure
- Moved to new Infrastructure solution to schedule ETL pipeline on Apache Airflow using Kubernetes Cluster with CI/CD pipelines deployment.
- Continuously working with stakeholder and Analyst to provide reliable and high quality data in AWS Redshift 
- Data Integration using REST API from various Tools Facebook, Google AdWords, Zendesk, CRM, Braze etc
- Optimised AWS Redshift cluster resource utilization and SQL queries tuning to improve the performance
- Parallelise ETL Jobs using Dask framework to improve the runtime performance of jobs
- Convert existing full copy data load ETLs to incremental load of legacy data pipeline
- Maintain infrastructure as a code data infrastructure in AWS stack using Cloudformation template
- Work in an agile using scrum framework

- Design and Implement Scalable Big Data Lambda Architecture Using Apache Spark, Cassandra and Elasticsearch for the Water management sector of Catalonia.
- Building Data Lake using Apache Cassandra Integrate different financial data sources i.e. SAP ISU, Oracle Informix. 
- Applied extensive data analytics using Apache Spark to calculate water consumption, bills, supplying entities.
- Elasticsearch used to persist the results for fast query results.

-Design and Implementation of Kappa Architecture using microservices. 
-Analyse mobility data pattern from live stream of data from various sensors.
-Build trajectories and Implement Markov Chain real-time udpated model to predict the next location of a moving object.
- Manage backend architecture for Big Data (Velocity and Volume) using Apache Kafka , Apache Cassandra and Apache Flink.
- Deployment of architecture into Amazon Web Service EC2 instances  for Apache Kafka and Cassandra and  Hadoop EMR for Apache Flink

Design & Develop web based business applications.
Responsible for different business process automation.
Design BI application technical solution based on business requirement and design.
Design ETL Procedures in SQL server integration services.
Create complex data solutions by leveraging database design, performance tuning, query optimization, indexes, data migration tools, and data analysis of back end data generated by products and services.
Provide support  to the clients for existing application.
Big Data Analysis: NPS reporting on millions of customer’s feedback record.
Projects: Mobilink’s Net Promoter System by Bain & Company -Customer Feedback Loop CFL, Franchise Portal,  ZCCM feedback portal

Design and develop Enterprise Level Automated Software for performance monitoring.
Develop KPIs Monitoring Dashboard web based for technical sub-departments.
Develop advance level SQL Queries (Bulk import, Merge, Cursor, Stored Procedures.) 
Project Documentation, Project scope, Architecture Diagrams.
Oracle and SQL Server database maintenance, backups & restore.
Big Data Analysis: Report generation on millions of subscriber’s data.
GSM Network data extraction and generate reports on different criteria.

Madiha Khalid🇩🇪