Vishal Mittal

Client: ABI InBev | Domain: CPG | Role: Data Engineer

Technologies Used: Azure, Azure Data Factory (ADF), Azure Databricks (ADB), Snowflake

Project: Migrated data from Snowflake to Azure Data Lake (Brewdat) and converted dataflows from ADF to Databricks using PySpark.

Architecture: Implemented Unity Catalog and Medallion Architecture with data transformation at the Silver layer.

Key Contributions:

• Created and optimized Databricks notebooks.
• Orchestrated data pipelines using Azure Data Factory.
• Ensured data quality through rigorous testing and validation.
• Worked in an agile framework for faster delivery.
• Collaborated with stakeholders for strategic planning and alignment.

Client: Mars | Domain: CPG | Role: Data Engineer

Technologies Used: Azure, Azure Data Factory (ADF), Azure Databricks (ADB), PostgreSQL

Project: Developed and enhanced Databricks notebooks to create Delta tables for the Price Pack Architecture (PPA), used by the Power BI team for sales dashboards.

Data Management: Stored ingested data in PostgreSQL alongside Delta tables for consumption by a web application.

Scaling: Expanded the project from a single-country to a multi-country deployment.

Key Contributions:

• Orchestrated data pipelines using Azure Data Factory.
• Enhanced data processing workflows and integration with reporting tools.
  
  

Client: Credit Suisse | Domain: Banking | Role: Data Engineer

Project Overview:
Developed an ingestion framework to migrate data from Cloudera (CDH) to Palantir Foundry (SFI), facilitating the transition from an on-premises platform to a cloud-based solution.

Key Responsibilities:

• Analyzed business requirements and designed the data ingestion framework.
• Developed Python scripts to trigger SQL scripts for data processing.
• Implemented data processing and transformation scripts using Python and SQL on Palantir Foundry.
• Managed daily data ingestion schedules for all datasets on Palantir Foundry.
• Processed daily and monthly data according to business logic, providing finalized data to the Tableau Dashboard team.
• Wrote transformation SQL scripts triggered by PySpark for data loading and transformation.
• Debugged issues and optimized solutions for improved performance.

Client: ABN-AMRO | Domain: Banking | Role: Big Data Engineer

Project Overview:
Developed an ingestion framework to migrate on-premises data from PostgreSQL and Oracle databases to Hive. This included both full and incremental loads through Spark jobs. Additionally, created tables using Spark SQL transformations and ingested final data into Elasticsearch for querying via Kibana.

Key Responsibilities:

• Designed the data ingestion framework based on business requirements.
• Developed Python scripts for data migration, including Hive queries and Sqoop commands.
• Captured metadata in separate tables to verify table counts.
• Utilized Python for data processing and transformation in Spark.
• Migrated data to HDFS using Sqoop and Spark, combined with Python scripts.
• Loaded and transformed complex data types and large datasets (e.g., XML, JSON, Parquet).
• Created and managed Hive tables (managed and external) and performed data loading.
• Maintained code in Git using Bitbucket.
• Scheduled jobs through Oozie.
• Developed SQL scripts, debugged issues, and optimized solutions.
• Processed data using Spark DataFrames.
• Participated in team building and training to prepare resources for the project.

Client: Allstate | Domain: Insurance |Role: Developer

Project Overview:
Worked on the Interlink project for rules, rating, and underwriting, supporting the second-largest personal lines property and casualty insurer in the U.S.

Key Responsibilities:

• Analyzed project and client requirements to design and implement solutions.
• Reviewed High-Level Design (HLD) and unit test plans, as well as unit test results.
• Managed the project through all SDLC phases, from development to production.
• Maintained quality artifacts and performed release and pre-production testing.
• Designed code based on state-specific underwriting guidelines.
• Provided ad-hoc reporting for actuarial queries and system corrective actions.
• Revised Auto/Property coverage rules and forms to compute premiums.
• Used tools: Universal Application, SQL Developer, Oracle, Beyond Compare.
• Worked in environments including C, Mainframe, Unix, and various development stages (Development, Functional, Integration, Staging, Pre-Staging, UAT, TPT, Production).