Yuriy is available for hire
Hire YuriyYuriy Margulis
Verified Expert in Engineering
Big Data Developer
Location
Los Angeles, CA, United States
Toptal Member Since
June 18, 2020
Yuriy是一名数据专家，在数据仓库方面拥有超过15年的经验, data engineering, feature engineering, big data, ETL/ELT, and business intelligence. As a big data architect and engineer, Yuriy specializes in AWS and Azure frameworks, Spark/PySpark, Databricks, Hive, Redshift, Snowflake, relational databases, tools like Fivetran, Airflow, DBT, Presto/Athena, and data DevOps frameworks and toolsets.
Portfolio
Databricks
Databricks
Paramount
Spark, PySpark, Scala, Databricks, Snowflake, Apache Airflow, SQL...
Crowd Consulting
Amazon Web Services (AWS), Data Warehouse Design, Data Warehousing...
Experience
Availability
Part-time
Preferred Environment
Amazon Web Services (AWS), Snowflake, Spark
The most amazing...
...我开发的项目是PriceGrabber数据仓库，经过多次平台变更，从5个主题增加到17个主题. I wrote multiple lines of SQL and other scripting c
Work Experience
Specialist Solutions Architect
2023 - PRESENT
Databricks
- 加入现场工程、通信、媒体和娱乐垂直部门.
- 为现场工程师、建筑师和客户提供技术支持.
- 执行数据仓库、数据工程、迁移和集成.
Technologies: Databricks
Senior Manager Data Engineering
2020 - 2023
Paramount
- 构建了一个收入数据集市，并向数据湖添加了一个服务器端主题区域.
- 管理一个团队并监督ETL监控、优化和性能调优.
- 代表数据工程团队参加公司架构协会活动.
Technologies: Spark, PySpark, Scala, Databricks, Snowflake, Apache Airflow, SQL, Amazon Web Services (AWS), Amazon Athena, Data Build Tool (dbt), Google BigQuery
Co-founder | CEO
2016 - 2023
Crowd Consulting
- Worked on full data warehouse implementations for multiple clients.
- Provided big data training and support for consulting partners.
- 使用AWS Kinesis为AWS S3数据仓库设计并构建ETL管道, Lambda, Hive, Presto, and Spark. The pipeline was written in Python.
- Delivered data warehouses, data lakes, data lakehouses, feature marts, BI systems, migrations, and integrations.
Technologies: Amazon Web Services (AWS), Data Warehouse Design, Data Warehousing, Amazon Athena, Tableau, Luigi, Scala, Python, Amazon S3 (AWS S3), Amazon DynamoDB, MySQL, PostgreSQL, Redshift, AWS Lambda, Apache Hive, Databricks, Spark, Hadoop, Amazon Elastic MapReduce (EMR)
Data Engineering Architect
2020 - 2020
CVS Health (via Toptal)
- ETL and feature engineering - personalization engine.
Technologies: RAPIDS, Scala, Python, Spark, Databricks, Azure
Data Engineer
2020 - 2020
Maisonette
- 使用Fivetran、dbt、Databricks搭建数据平台和数据湖.
- Participated in the development of a BI platform in Looker.
- Performed CI/CD deployment and operational support.
Technologies: Amazon Web Services (AWS), Fivetran, Looker, Python, Apache Airflow, Snowflake, PostgreSQL
Data Engineer
2019 - 2020
Teespring (via Toptal)
- 将数据仓库ETL管道从Airflow/Redshift迁移到Fivetran、Databricks和Snowflake.
Technologies: Amazon Web Services (AWS), APIs, Redshift, Apache Airflow, Python, Spark, Snowflake, Databricks, Fivetran
Data Engineer
2018 - 2019
BCG GAMMA (via Toptal, Three Contracts)
- Provided engineering support for data scientists.
- 在AWS S3中设计并构建了特色工程数据集市和客户360°数据湖.
- Designed and developed a dynamic S3-to-S3 ETL system in Spark and Hive.
- Completed various DevOps tasks included an Airflow installation, development of Ansible playbooks, and history backloads.
- 参与一个特征工程项目，涉及Hortonworks、Spark、Python、Hive和Airflow.
- 在Microsoft Azure和Databricks上使用PySpark构建一对一的营销功能工程管道(使用ADF), ADL, Databricks Delta Lake, and ADW as a source).
Technologies: Ansible, Boto 3, Apache Airflow, PostgreSQL, Relational Database Services (RDS), AWS Glue, Amazon Athena, Presto DB, Apache Hive, Spark, Python
Vice President, Data
2017 - 2018
Enervee
- Managed the data engineering, BI reporting, and data science teams.
- Worked as a hands-on data engineer.
- Built a data lake on AWS.
- Developed a reporting system with Redash/Presto.
Technologies: Amazon Web Services (AWS), Redash, Apache Airflow, Python, Amazon S3 (AWS S3), Amazon Aurora, MySQL, PostgreSQL, Redshift, Apache Hive, Presto DB, Spark, Amazon Elastic MapReduce (EMR), Hadoop
Big Data Architect
2016 - 2017
ITG
- 全职工作，作为交易成本分析系统的数据架构师.
- 在ITG的私有云上安装四节点Apache Hadoop/Spark集群.
- 将Apache Spark技术嵌入ITG数据平台，进行平台POC.
- Supported the development of a platform POC for Kx Kdb+; also converted Sybase IQ queries to Kdb+ Q language.
技术:Q, Kdb+， Informatica, Sybase, Python, Spark, Apache Hive, Hadoop
Data Engineer
2016 - 2017
American Taekwondo Association (via Toptal)
- 将数据从旧的Oracle数据库转换到新设计的SQL Server数据库.
- Wrote SQL scripts, stored procedures, kettle transformations.
- Administered two databases.
- Performed extensive data cleansing and validation.
Technologies: Pentaho, Oracle, SQL
Director, Data Warehouse
2015 - 2016
Connexity
- 为PriceGrabber和Shopzilla管理两个数据仓库和BI团队. Connexity is also known as PriceGrabber, Shopzilla, and BizRate.
- Handled operational support for the PriceGrabber data warehouse. Recovered data warehouse after the data center migration.
- Merged one data warehouse into another and retired one of them. Hands-on designed business and data integration architecture; developed data validation scripts and ETL integration code. 管理BI报告系统从Cognos到OBIEE和Tableau的转换.
- 定义了组合数据仓库的技术平台变更策略.
- 为ETL和数据验证创建了SQL: PL SQL存储过程、包和匿名脚本.
- Completed an Amazon Redshift project.
- Worked on and completed a Cloudera Impala project.
Technologies: Amazon Web Services (AWS), Linux, Python, Perl, Tableau, Oracle Business Intelligence Enterprise Edition 11g (OBIEE), Cognos 10, Impala, Hadoop, Redshift, PL/SQL, Oracle
Director, Data Warehouse
2008 - 2015
PriceGrabber
- Oversaw the company's data services, defined the overall and technical strategy for data warehousing, business intelligence, and big data environments.
- 雇佣并管理一支混合陆上(美国)/海上(印度)工程团队.
- 将数据仓库重新平台化为Oracle Exadata X3/Oracle ZFS组合, 在数据仓库环境中增加了大数据和机器学习组件.
- 支持24x7x365全天候运营，符合公司顶级生产SLA.
- Wrote thousands of lines of PL/SQL, PL/pgSQL, MySQL, and HiveQL code.
- 在Kettle内部用Perl、Python和JavaScript编写ETL脚本.
- 在多个项目中使用大数据(Hadoop, Pig, Hive, Mahaut).
- 为一个Pentaho (Kettle) CE ETL重新设计项目开发了一个基于工具的ETL.
- 为各种类型的项目(Python, SciPy, NumPy和Pandas)从事机器学习工作。.
Technologies: Pentaho, Linux, Python, Perl, MySQL, PostgreSQL, Apache Hive, Apache Pig, Hadoop, Oracle
Director, Data Warehouse
2007 - 2008
Edmunds
- Managed a data warehouse team and project pipeline; supported operations.
- 为ETL和数据验证创建PL/SQL存储过程、包和匿名脚本.
- Worked on a tool-based ETL for multiple Informatica projects.
Technologies: Linux, Perl, Informatica, Oracle
Manager, Data Warehouse
2003 - 2007
Universal Music Group
- Managed, developed, and operated a CRM data warehouse.
- Wrote PL/SQL, MySQL, and Perl code.
- Administered to a Cognos reporting system.
- 为OLAP报告系统的多个支持项目使用c#.
- Designed and developed a MSAS OLAP cube system.
技术:Linux, Perl, c#， Cognos 10, MySQL, Microsoft SQL Server, Oracle
Director, Decision Support and Financial Systems
2001 - 2003
MediaLive International
- Managed a data warehouse, BI, and CRM systems.
- Assumed responsibilities over an Oracle EBS application team.
- 为数据仓库ETL和Oracle应用程序集成开发了PL/SQL代码.
- 使用SQL server完成多个Transact-SQL和分析服务项目.
- Worked on a tool-based ETL for multiple epiphany EPI*Channel projects.
Technologies: Unix, VB, Microsoft SQL Server, Oracle EBS, Oracle
Senior Principal Consultant (Professional Services, Essbase Practice)
1999 - 2001
Hyperion (Currently: Oracle)
- Led a practice for a consulting company covering for multiple clients.
- 开发Essbase卫星系统:关系数据仓库和数据集市, reporting systems, ETL systems, CRM's, EPP's, ETL in and out of Essbase and with Essbase itself.
- 通过为团队的Oracle项目管道提供全面支持，参与了多个PL/SQL项目.
- 帮助为多个Transact-SQL和分析服务项目开发SQL服务器.
- Developed a tool-based ETL for an Informatica project.
- 与Hyperion, Essbase, Enterprise, Pillar，规划，财务分析和VBA项目合作.
Technologies: Essbase, Hyperion, Informatica, Visual Basic for Applications (VBA), Microsoft SQL Server, Oracle
Skills
Languages
Python, SQL, PL/pgSQL, Snowflake, c#， Visual Basic for Applications (VBA)， VB, Scala, Q, Perl
Frameworks
Apache Spark, Spark, Presto DB, Hadoop
Tools
Apache Airflow, Amazon Elastic MapReduce (EMR), Amazon Athena, Pentaho Data Integration (Kettle), AWS Glue, Impala, Oracle Business Intelligence Enterprise Edition 11g (OBIEE), Tableau, Hyperion, Redash, Boto 3, Ansible, Looker, Informatica PowerCenter
Paradigms
ETL, Business Intelligence (BI), Management, Database Design, Testing
Platforms
Oracle, Databricks, Azure, Linux, Apache Pig, Pentaho, Unix, AWS Lambda, Amazon Web Services (AWS)
Storage
PostgreSQL, Apache Hive, Databases, Oracle PL/SQL, Redshift, Microsoft SQL Server, MySQL, PL/SQL, Amazon DynamoDB, Amazon S3 (AWS S3), Sybase, Kdb+, Amazon Aurora, Cassandra, Essbase, Data Lakes, Data Pipelines
Other
数据仓库，数据架构，领导力，团队指导，技术战略 & Architecture, Big Data, Software Development, Fivetran, Data Warehouse Design, Snowpark, Informatica, Oracle EBS, Relational Database Services (RDS), APIs, perlpod, Unix Shell Scripting, MSAS, Cognos 10, Data Build Tool (dbt), Google BigQuery, Security, Deployment, Data Modeling
Libraries/APIs
Luigi, RAPIDS, PySpark
Education
2016 - 2016
Apache Spark数据科学与工程结业证书
UC BerkeleyX(伯克利在线课程)-加州伯克利(美国)
2012 - 2012
Cloudera Apache Hadoop开发人员培训结业证书
Cloudera University - New York, New York (USA)
1995 - 1995
Certificate of Completion in Oracle Database Administration
UCI Extension - Irvine, California (USA)
1975 - 1980
Diploma (Master of Science Equivalent) Degree in Applied Mathematics
Odessa I.I. Mechnikov University - Odessa, Ukraine
Certifications
JUNE 2023 - PRESENT
Databricks Certified Data Engineer Professional
Databricks
Collaboration That Works
How to Work with Toptal
在数小时内，而不是数周或数月，我们的网络将为您直接匹配全球行业专家.
1
Share your needs
在与Toptal领域专家的电话中讨论您的需求并细化您的范围.
2
Choose your talent
在24小时内获得专业匹配人才的简短列表，以进行审查，面试和选择.
3
Start your risk-free talent trial
Work with your chosen talent on a trial basis for up to two weeks. Pay only if you decide to hire them.
Top talent is in high demand.
Start hiring