Lsf-join: Locality Sensitive Filtering For Distributed All-pairs Set Similarity Under Skew

Abstract

All-pairs set similarity is a widely used data mining task, even for large and high-dimensional datasets. Traditionally, similarity search has focused on discovering very similar pairs, for which a variety of efficient algorithms are known. However, recent work highlights the importance of finding pairs of sets with relatively small intersection sizes. For example, in a recommender system, two users may be alike even though their interests only overlap on a small percentage of items. In such systems, some dimensions are often highly skewed because they are very popular. Together these two properties render previous approaches infeasible for large input sizes. To address this problem, we present a new distributed algorithm, LSF-Join, for approximate all-pairs set similarity. The core of our algorithm is a randomized selection procedure based on Locality Sensitive Filtering. Our method deviates from prior approximate algorithms, which are based on Locality Sensitive Hashing. Theoreticall

Stats

• citations6
• S2 citations—
• github stars0
• HF likes0
• heat score6.34
• arxiv keyrashtchian2020lsf

Related papers

• Improving Distributed Similarity Join In Metric Space With Error-bounded Sampling (2019)0.00
• Locality Sensitive Hashing For Set-queries, Motivated By Group Recommendations (2020)0.00
• A Framework For Similarity Search With Space-time Tradeoffs Using Locality-sensitive Filtering (2016)8.35
• Subsets And Supermajorities: Optimal Hashing-based Set Similarity Search (2019)5.84
• Xling: A Learned Filter Framework For Accelerating High-dimensional Approximate Similarity Join (2024)0.00
• Efficient Similarity Search In Dynamic Data Streams (2016)0.00
• Unconventional Application Of K-means For Distributed Approximate Similarity Search (2022)5.84
• Massively-parallel Similarity Join, Edge-isoperimetry, And Distance Correlations On The Hypercube (2016)2.26