Executive Summary
The Global Professional Services Enterprise’s operates the leading expert network, requiring precise, contextualized data to connect clients with experts quickly and compliantly. LakeFusion provides The Global Professional Services Enterprise with a highly tailored, business-driven single view of company entities by leveraging advanced AI matching within their existing Databricks Lakehouse, enabling better search efficiency, improved analytics, and critical risk mitigation.
Key Outcomes:
- Established a Golden Record Foundation: Mastered company records across CapIQ, AggK, and internal CRM sources, creating a single source of truth for over 3 million source corporate entities.
- Enabled Business-Driven Granularity: Successfully tuned the matching logic using custom LLM prompts to define and consolidate entities based on the Global Professional Services Enterprise’s flexible operational requirements, not just strict legal definitions.
- Optimized Compute Strategy: Built and deployed a rules-based matching engine to supplement and augment LLM-based matching strategy
Business Challenge
The Global Professional Services Enterprise scale and reliance on multiple disparate data sources led to significant data fragmentation and quality issues:
- Fragmented Company Data: Company information was siloed across internal systems (CRM_Cadvisors) and external feeds (CapIQ, AggK), leading to duplicative entries, conflicting attributes (e.g., employee count, revenue), and unreliable analytics.
- High Record Count: High record counts and a mix of ambiguous cases requiring complex AI-powered evaluation with many “easy” matches led to development of LakeFusion’s rules-based matching engine to balance performance and cost-consciousness with a need for AI-powered tooling.
Solution Overview
MDM Approach
LakeFusion was implemented directly on the Global Professional Services Enterprise’s Lakehouse, aligning with their Medallion architecture (Bronze, Silver, Gold).
After ingesting and flattening data from CapIQ, AggK, and CRM_Cadvisors, LakeFusion utilized its AI-first entity resolution engine paired with custom-tuned LLM prompts to intelligently group and match company records.
The solution focused heavily on iterative prompt refinement to explicitly instruct the LLM the desired granularity—distinguishing between parent companies and operational subsidiaries—ensuring the resulting master data met the Global Professional Services Enterprise’s unique business needs.
Implementation Components
- Data Profiling & Assessment - Initial LakeFusion runs documented issues related to name variance and provided insight into attribute overlap between CapIQ and AggK, leading to tailored survivorship rules.
- Matching & Linking Strategy - Implemented a sophisticated matching strategy using Databricks Vector Search and tuned LLMs (DeepSeek R1 and Llama 3 were benchmarked against Llama 4 Maverick) guided by a custom prompt to enforce the Global Professional Services Enterprise specific entity granularity.
- Golden Record Creation - Finalized the gp_company_master entity schema and configured detailed survivorship logic based on documented column priority rules, resolving conflicts (e.g., employee count, primary name).
- Data Quality & Governance - Developed and integrated custom logic to address key operational needs, including the ability to handle upstream delete flags by triggering an automated "force unmatch" operation on the golden record.
- Integration Enablement - Established the framework for utilizing Change Data Feed (CDF) on source systems (inputs) and the master data output (Gold layer) to enable dynamic, near real-time data flow for downstream consumption.
Business Value Delivered
Primary Outcome
The establishment of the single gp_company_master repository transformed the Global Professional Services Enterprise’s ability to conduct efficient and accurate operational workflows:
- Before MDM: Search and matching workflows struggled with ambiguity, frequently merging distinct parent/subsidiary entities, requiring significant manual verification, and providing inconsistent data for financial analysis (e.g., LTM Revenue).
- After MDM: Comprehensive company profiles tailored to the Global Professional Services Enterprise’s desired granularity enabled better target identification, streamlined expert matching, and provided reliable, unified attributes (like address, name, employee count) for internal systems.
Secondary Business Benefits
- Compliance & Risk - Established the foundation to implement Person MDM, which will be critical to consolidating records and ensuring 100% adherence to DNC flags, mitigating significant compliance and reputational risk.
- Data Architecture - Validated the Medallion Lakehouse pattern for mastering, proving that LakeFusion could operate natively within their Databricks environment while providing crucial cost transparency via improved asset tagging.
- Future Scalability - The successful refinement of LLM prompts and model selection ensures that future large-scale production runs (processing millions of records) can be executed cost-effectively and predictably using Provisioned Throughput.
- Operational Efficiency - Reduced manual data reconciliation efforts by automatically resolving attribute conflicts and providing a unified view, improving staff productivity.
Key Success Factors
Several factors were critical to achieving project objectives and validating the LakeFusion approach:
- Iterative LLM Prompt Tuning: The sustained effort to refine the custom prompt was paramount in successfully instructing the LLM Global Professional Services Enterprise’s desired level of entity granularity, which was a unique business requirement.
- Data Gravity Alignment: Leveraging the existing Databricks Lakehouse architecture maximized performance and simplified data governance via Unity Catalog, drastically reducing data movement complexity and cost.
- Clear Feature Focus: Prioritizing the development of critical governance features (like automated handling of upstream delete flags/force unmatch) ensured that the solution addressed non-standard operational requirements upfront.
- Cost-Performance Optimization: The detailed cost analysis contrasting Llama 4 Maverick and DeepSeek R1, combined with the discovery of the non-scale-to-zero issue, led to a pragmatic and cost-effective production model choice (Llama 3 with enhanced prompts).
Best Practices
- Use deterministic matching to help reduce overall costs and accelerate match processing
- Use match maven experiments with subsets of data to expedite the match “tuning” process by identifying patterns quickly
- For companies maturing their data practice, prioritizing a domain with public data will expedite MDM adoption
- Heavy reliance on extensive prompts can cause higher consumption costs and result in LLM hallucinations
Conclusion
The Company Data MDM pilot implementation at the Global Professional Services Enterprise, leveraging LakeFusion, successfully established a robust data foundation capable of handling the client’s unique, complex entity granularity requirements. By combining Databricks’ native power with LakeFusion’s AI-first matching, the project delivered a reliable source of truth for corporate entities and laid the essential groundwork for expanding into the Person domain to tackle high-priority compliance risks associated with DNC flags.
This initiative established technical patterns and governance capabilities that position the Global Professional Services Enterprise for continuous data quality improvement and measurable impact on operational efficiency and risk mitigation.
Stay Ahead in Enterprise Data
Insights on master data management, Databricks, and building AI-ready data platforms—delivered occasionally, without the noise.
