Introduction to Data Engineering
- Explore the role of a data engineer
- Analyze data engineering challenges
- Introduction to BigQuery
- Data lakes and data warehouses
- Transactional databases versus data warehouses
- Partner effectively with other data teams
- Manage data access and governance
- Build production-ready pipelines
- Review Google Cloud customer case study
Building a Data Lake
- Introduction to data lakes
- Data storage and ETL options on Google Cloud
- Building a data lake using Cloud Storage
- Securing Cloud Storage
- Storing all sorts of data types
- Cloud SQL as a relational data lake
Building a Data Warehouse
- The modern data warehouse
- Introduction to BigQuery
- Getting started with BigQuery
- Loading data
- Exploring schemas
- Schema design
- Nested and repeated fields
- Optimizing with partitioning and clustering
Introduction to Building Batch Data Pipelines
- EL, ELT, ETL
- Quality considerations
- How to carry out operations in BigQuery
- Shortcomings
- ETL to solve data quality issues
Executing Spark on Dataproc
- The Hadoop ecosystem
- Run Hadoop on Dataproc
- Cloud Storage instead of HDFS
- Optimize Dataproc
Serverless Data Processing with Dataflow
- Introduction to Dataflow
- Why customers value Dataflow
- Dataflow pipelines
- Aggregating with GroupByKey and Combine
- Side inputs and windows
- Dataflow templates
- Dataflow SQL
Manage Data Pipelines with Cloud Data Fusion and Cloud Composer
- Building batch data pipelines visually with Cloud Data Fusion
- Components
- UI overview
- Building a pipeline
- Exploring data using Wrangler
- Orchestrating work between Google Cloud services with Cloud Composer
- Apache Airflow environment
- DAGs and operators
- Workflow scheduling
- Monitoring and logging
Introduction to Processing Streaming Data
Serverless Messaging with Pub/Sub
- Introduction to Pub/Sub
- Pub/Sub push versus pull
- Publishing with Pub/Sub code
Dataflow Streaming Features
- Steaming data challenges
- Dataflow windowing
High-Throughput BigQuery and Bigtable Streaming Features
- Streaming into BigQuery and visualizing results
- High-throughput streaming with Cloud Bigtable
- Optimizing Cloud Bigtable performance
Advanced BigQuery Functionality and Performance
- Analytic window functions
- Use With clauses
- GIS functions
- Performance considerations
Introduction to Analytics and AI
- What is AI?
- From ad-hoc data analysis to data-driven decisions
- Options for ML models on Google Cloud
Prebuilt ML Model APIs for Unstructured Data
- Unstructured data is hard
- ML APIs for enriching data
Big Data Analytics with Notebooks
- What’s a notebook?
- BigQuery magic and ties to Pandas
Production ML Pipelines
- Ways to do ML on Google Cloud
Vertex AI Pipelines
- AI Hub
Custom Model Building with SQL in BigQuery ML
- BigQuery ML for quick model building
- Supported models
Custom Model Building with AutoML
- Why AutoML?
- AutoML Vision
- AutoML NLP
- AutoML tables
