Query-Based Sampling: Can we do Better than Random?

A.S. Tigelaar, Djoerd Hiemstra

Research output: Book/ReportReportProfessional

10 Downloads (Pure)

Abstract

Many servers on the web offer content that is only accessible via a search interface. These are part of the deep web. Using conventional crawling to index the content of these remote servers is impossible without some form of cooperation. Query-based sampling provides an alternative to crawling requiring no cooperation beyond a basic search interface. In this approach, conventionally, random queries are sent to a server to obtain a sample of documents of the underlying collection. The sample represents the entire server content. This representation is called a resource description. In this research we explore if better resource descriptions can be obtained by using alternative query construction strategies. The results indicate that randomly choosing queries from the vocabulary of sampled documents is indeed a good strategy. However, we show that, when sampling a large collection, using the least frequent terms in the sample yields a better resource description than using randomly chosen terms.
Original languageUndefined
Place of PublicationEnschede
PublisherCentre for Telematics and Information Technology (CTIT)
Number of pages8
Publication statusPublished - 1 Feb 2010

Publication series

NameCTIT Technical Report Series
No.TR-CTIT-10-04
ISSN (Print)1381-3625

Keywords

  • METIS-270727
  • CR-H.3.3
  • CR-H.3.4
  • query-based sampling
  • EWI-17404
  • Distributed Information Retrieval

Cite this

Tigelaar, A. S., & Hiemstra, D. (2010). Query-Based Sampling: Can we do Better than Random? (CTIT Technical Report Series; No. TR-CTIT-10-04). Enschede: Centre for Telematics and Information Technology (CTIT).