Dataset Discovery In Data Lakes

Abstract

Data analytics stands to benefit from the increasing availability of datasets that are held without their conceptual relationships being explicitly known. When collected, these datasets form a data lake from which, by processes like data wrangling, specific target datasets can be constructed that enable value-adding analytics. Given the potential vastness of such data lakes, the issue arises of how to pull out of the lake those datasets that might contribute to wrangling out a given target. We refer to this as the problem of dataset discovery in data lakes and this paper contributes an effective and efficient solution to it. Our approach uses features of the values in a dataset to construct hash-based indexes that map those features into a uniform distance space. This makes it possible to define similarity distances between features and to take those distances as measurements of relatedness w.r.t. a target table. Given the latter (and exemplar tuples), our approach returns the most rel

Stats

• citations108
• S2 citations—
• github stars0
• HF likes0
• heat score15.28
• arxiv keybogatu2020dataset

Related papers

• Pylon: Semantic Table Union Search In Data Lakes (2023)0.00
• MQRLD: A Multimodal Data Retrieval Platform With Query-aware Feature Representation And Learned Index Based On Data Lake (2024)8.35
• Deepjoin: Joinable Table Discovery With Pre-trained Language Models (2022)12.25
• Hierarchical Locality Sensitive Hashing For Structured Data: A Survey (2022)0.00
• An Instance Selection Algorithm For Big Data In High Imbalanced Datasets Based On LSH (2022)0.00
• The Lernaean Hydra Of Data Series Similarity Search: An Experimental Evaluation Of The State Of The Art (2020)0.00
• Qwlsh: Cache-conscious Indexing For Processing Similarity Search Query Workloads In High-dimensional Spaces (2019)4.52
• Modeltables: A Corpus Of Tables About Models (2025)2.35