Challenge 6: Preparing the Context Layer

Previous Challenge Next Challenge

Target Persona: Data Engineer / AI Architect Estimated Duration: 40 minutes

Introduction

Before creating your conversational AI agents, you need to build a centralized context layer. In Google Cloud, Knowledge Catalog allows you to store technical and business metadata, making it well suited for this purpose. This layer ensures your agents understand your business terminology, operational metadata, and data asset structures, leading to higher accuracy and fewer hallucinations.

Task 6.1: Technical Metadata Enrichment

  1. Navigate to your BigQuery dataset in BigQuery Studio. Due to the integration between the two services, when you add metadata in BigQuery, it is also visible in Knowledge Catalog.
  2. Enrich the table disneyland_reviews by adding schema descriptions to the columns.
  3. Add clear technical metadata descriptions to your vector embedding, image analysis JSON, and source URI columns across the various tables.

Task 6.2: Business Glossary Alignment

Conversational agents often struggle when user prompts use everyday business terminology (e.g., “frequent visitor”) that does not match physical database column names. Using Dataplex Business Glossaries, you will create a unified semantic layer that bridges business definitions directly to both analytical engines (BigQuery) and transactional databases (AlloyDB).

  1. Glossary Creation: Create a centralized Business Glossary in Knowledge Catalog.
  2. Core Definitions: Define core business terms and definitions inside the glossary” define the terms “Rollercoaster”, “Premium visitor”, & “Buffet Dining Category”
    An example: “Premium visitor”: A visitor who left more than 2 reviews).
  3. Asset Mapping to BigQuery: Link these business terms directly to their corresponding BigQuery columns to map technical metadata to business language.
  4. Asset Mapping to AlloyDB: Try mapping some terms to AlloyDB assets as well to see how Knowledge Catalog equally integrates to operational & analytical databases.

Task 6.3: Automated profiling & quality

Feeding unvalidated or drifting data into an AI context window directly causes hallucinated answers and unreliable agent behavior. Dataplex Auto Data Quality & Data Profiling automates the inspection of data distributions, custom business logic, and more. By running automated profiling and enforcing quality assertions before data reaches the retrieval layer, you protect downstream AI agents from faulty context and ensure every generated insight is backed by verified data.

  1. Run a data profile scan on the disneyland_reviews table. Analyze the results.
  2. Define two data quality rules (rating = [1-5], reviewer_location != NULL) and run a Data Quality Scan to ensure the reviews table matches these criteria.

Task 6.4: Automated GCS Metadata Generation

Sometimes there is too much data and not enough humans to enrich your data. BigQuery offers automatic discovery and metadata generation for data in Cloud Storage, such as our PDFs and images. Set up an automated extraction pipeline to handle documentation and unstructured assets within your environment. This turns raw object storage into an intelligent, searchable document repository that AI agents can effortlessly discover and retrieve.

 Tip This can be done in the BigQuery Metadata Curation tab.

  1. Configure the pipeline to analyze your unstructured gs://ghacks-disneyland-on-gcp/ bucket and generate metadata automatically.

Success Criteria

To validate this challenge, you must demonstrate the following:

  • Show the enriched schema descriptions for your target tables directly within the BigQuery Console.
  • Provide a summary or export of the linked terms inside your centralized Disneyland Business Glossary.
  • Show the successful pipeline logs or sample metadata tags generated for the PDF assets in Cloud Storage.

Previous Challenge Next Challenge