Content ownership and usage of crawler: The ownership of the content crawled belongs to the initial publishers. The owner of the crawler (or receiver of this feed) can only utilize the feeds for indexing, analysis and integrated searches – not for re-publishing or sharing full-text content.
Any crawling and indexing of open web is subject to acceptance from the site – normally communicated through robot txt file. As long as not stated in robot.txt, the sites normally allow crawling and indexing for the purposes as described above. Our crawling practices shall respect any requirements about crawling not desired. In some regions the online media has established license agreements for monitoring feeds and crawlers in order to regulate this activity – such as NLA in UK and Klareringssentralen in Norway. Contact us for further advice on the use of crawling & feeds in these countries.