Designing A General Deep Web Harvester by Harvestability Factor

Mohammadreza Khelghati, Maurice van Keulen, Djoerd Hiemstra

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

77 Downloads (Pure)


To make deep web data accessible, harvesters have a crucial role. Targeting different domains and websites enhances the need of a general-purpose harvester which can be applied to different settings and situations. To develop such a harvester, a large number of issues should be addressed. To have all influential elements in one big picture, a new concept, called harvestability factor (HF), is introduced in this paper. The HF is defined as an attribute of a website (HFW) or a harvester (HF_H) representing the extent to which the website can be harvested or the harvester can harvest. The comprising elements of these factors are different websites’ or harvesters’ features. These elements are gathered from literature or introduced through the authors’ experiments. In addition to enabling designers of evaluating where they products stand from the harvesting perspective, the HF can act as a framework for designing harvesters. Designers can define the list of features and prioritize their implementations. To validate the effectiveness of HF in practice, it is shown how the HFs0 websites and how this is useful in designing a harvester. To validate the HF H as an evaluation metric, it is shown how it can be calculated for the harvester implemented by the authors. The results show that the developed harvester works pretty well for the targeted test set by a score of 14.783 of 15.
Original languageUndefined
Title of host publicationProceedings of the international ISWC workshop on Surfacing the Deep and the Social Web, SDSW 2014
Place of PublicationAachen, Germany
Number of pages16
Publication statusPublished - 19 Oct 2014
EventInternational ISWC Workshop on Surfacing the Deep and the Social Web, SDSW 2014 - Riva Del Garda, Italy
Duration: 19 Oct 201423 Oct 2014

Publication series

NameCEUR Workshop Proceedings
PublisherCEUR Workshop Proceedings
ISSN (Print)1613-0073


WorkshopInternational ISWC Workshop on Surfacing the Deep and the Social Web, SDSW 2014
Other19-23 October 2014


  • EWI-25355
  • METIS-309685
  • IR-93326
  • CR-H.3.3

Cite this