Hafiz Hamid，美国加州旧金山的开发人员
Hafiz is available for hire
Hire Hafiz

Hafiz Hamid

Verified Expert  in Engineering

Web Scraping Developer

Location
San Francisco, CA, United States
Toptal Member Since
June 18, 2020

Hafiz是一位经验丰富的软件架构师，在Bing(微软)等公司领导了12年的复杂软件项目。, Lyft, and Salesforce.现在，他正在追求一份自由职业. His areas of expertise are back-end/server development, databases, big data, cloud computing, DevOps, web crawling, and search engines.

Web ScrapingData Warehouse DesignData WarehousingBig DataBig Data ArchitectureDatabasesSQLPythonDevOpsHadoopAmazon EC2Amazon Web Services (AWS)Amazon DynamoDBAmazon CloudWatchAWS LambdaRedshiftPub/SubxQueryAWS CloudWatchAWS KinesisHiveLucene

Portfolio

Lyft, Inc.
Hadoop, Apache Flink, Apache Kafka, AWS云架构，Amazon DynamoDB...
Salesforce.com
Apache Lucene, Apache Solr, Java
Microsoft (Bing Search)
机器学习，Apache Hive, Hadoop, Microsoft SQL Server, c#， .NET

Experience

SQL - 10 yearsWeb Scraping - 6 yearsPython - 5 yearsLarge Scale Distributed Systems - 5 yearsBig Data Architecture - 5 yearsDevOps - 4 yearsAmazon Web Services (AWS) - 3 yearsPub/Sub - 3 years

Availability

Part-time

Preferred Environment

Git, Linux, MacOS

The most amazing...

...我在Lyft建立了一个实时流数据管道. 我做了一个网络爬虫每天在必应上抓取10亿个网页.com.

Work Experience

Staff Software Engineer (Full-time)

2015 - 2018
Lyft, Inc.
  • Worked as the tech lead and architect on streaming platform team; also drove the vision and strategy.
  • 为Lyft构建实时事件摄取和发布/订阅基础设施，每天摄取/移动超过2000亿个事件.
  • 在Lyft开发了高度可扩展和可靠的消息总线，用于数百个内部微服务之间的异步通信.
  • 维护多个0层业务，提供5个9级的可靠性保证/SLA.
  • Trained and mentored dozens of other engineers.
Technologies: Hadoop, Apache Flink, Apache Kafka, AWS云架构，Amazon DynamoDB, Amazon Kinesis, Amazon CloudWatch, Redshift, Amazon S3 (AWS S3), Amazon Simple Queue Service (SQS), AWS Lambda, Amazon EC2, Python

Principal Member of Technical Staff (Full-time)

2014 - 2015
Salesforce.com
  • 开发了几个相关特性，包括根据Salesforce的需求定制Apache Lucene的评分框架.
  • 实现的基础设施工作支持运行时特征提取，用于训练基于ml的排名器，并将其集成到Apache Solr的查询处理管道中.
  • 设计搜索基础架构，将Salesforce搜索的静态排名特性扩展到100%的文档(由于基础架构的限制，目前只能部分启用).
Technologies: Apache Lucene, Apache Solr, Java

Senior Software Engineer (Full-time)

2005 - 2014
Microsoft (Bing Search)
  • 领导工程师团队为分布式网络爬虫和内容提取平台开发可扩展的基础设施，使其能够每天从数百个网站(如亚马逊)抓取数亿个网络文档.com, Imdb.com, Walmart.com)，并对其进行解析，提取结构化内容，以丰富必应的搜索索引.
  • 因上述项目获得微软金星奖.
  • Developed a log mining platform to enrich a local search index; enabled it to algorithmically discover/mine URLs and search keywords, associated with local businesses (restaurants, hotels, banks, etc.)，通过挖掘搜索结果点击日志(pb级数据). 该平台已在20多个必应市场使用，以丰富本地搜索索引，并缩小与谷歌的URL覆盖率差距.
  • 作为技术领导和集成电路能力，将基于机器学习的文本分类框架(最初由微软研究院构思)增强和发展为分类平台，并将其与本地数据管道集成.
  • Developed a process to train, 评估和使用统计模型，这些模型将全球数亿本地企业分类为1个以上的分类法,000 categories; for the above project.
  • 管理(从技术领先的角度来看)本地数据摄取/处理管道的日常维护和操作，该管道提供给Bing本地搜索引擎的索引.
  • 负责必应娱乐搜索(音乐)的后端数据采集/处理管道, movies, TV shows, and more).
技术:机器学习，Apache Hive, Hadoop, Microsoft SQL Server, c#， .NET

Professional Services Consultant (Full-time)

2005 - 2006
Teradata Corporation
  • Developed automated ETL framework, 用于DHL (Teradata客户)，以便它从多个异构源摄取数据并集成到企业数据仓库中.
  • 领导Eircom元数据驱动ETL工具项目的四人开发团队，该项目旨在开发通用解析和转换引擎，用于从50多种不同的半结构化CDR格式中提取数据. (Eircom是爱尔兰领先的电信运营商).
  • 为新员工进行Teradata培训和数据仓库研讨会.
Technologies: Teradata, Java, SQL

OpenSecrets.org Scraper

这个Python程序利用开源的Scrapy框架来抓取OpenSecrets.org for campaign contribution data. By providing a company or entity name as input (e.g., "Disney"), scraper下载有关该实体在过去25年中对美国国家和地方选举活动的贡献的信息.

Lyft, Inc.

Technologies: Python, AWS Cloud (EC2, Lambda, Kinesis, DynamoDB, SQS, S3, Redshift, CloudWatch), Apache Kafka, Apache Flink, Hadoop/Hive

Salesforce.com

技术:Java, Apache Solr/Lucene, Search relevance

Microsoft (Bing Search)

我研究了这个网络抓取和提取框架.

Technologies: C#/.. NET, Microsoft SQL Server, Hadoop/Hive，机器学习

Teradata Corporation

Technologies: SQL, Java, Teradata
2009 - 2011

Master's Degree in Computer Science and Engineering

University of Washington - Seattle, WA, USA

2001 - 2005

Bachelor's Degree in Computer Science

FAST |巴基斯坦伊斯兰堡国立计算机与新兴科学大学

JANUARY 2006 - PRESENT

Teradata Certified Master

Teradata Corporation

Libraries/APIs

Apache Lucene

Tools

Amazon Simple Queue Service (SQS)， Amazon CloudWatch, Zapier, Apache Solr, Git, Flink

Frameworks

Hadoop, Scrapy, Flask, .NET, Django

Paradigms

DevOps, ETL, Agile Software Development

Languages

Python, SQL, C#.NET, Java, HTML, XQuery, XML, XPath, C#, JavaScript

Platforms

AWS Lambda, Amazon EC2, Amazon Web Services (AWS)， Apache Kafka, Apache Flink, MacOS, Linux

Storage

Amazon DynamoDB, PostgreSQL, Redshift, Amazon S3 (AWS S3), Databases, Teradata, SQL Server 2010, Apache Hive, Elasticsearch, Microsoft SQL Server

Other

Data Warehouse Design, Web Scraping, Data Warehousing, Amazon Kinesis Data Firehose, Big Data, Amazon Kinesis, Big Data Architecture, Stream Processing, Large Scale Distributed Systems, Pub/Sub, Machine Learning, Search Engine Development, Information Retrieval, Data Modeling, Text Classification, AWS Cloud Architecture

Collaboration That Works

How to Work with Toptal

在数小时内，而不是数周或数月，我们的网络将为您直接匹配全球行业专家.

1

Share your needs

在与Toptal领域专家的电话中讨论您的需求并细化您的范围.
2

Choose your talent

在24小时内获得专业匹配人才的简短列表，以进行审查，面试和选择.
3

Start your risk-free talent trial

与你选择的人才一起工作，试用最多两周. Pay only if you decide to hire them.

Top talent is in high demand.

Start hiring