Link Details

MongoDB was used early on at Scrapinghub to store scraped data because it’s convenient. Scraped data is represented as (possibly nested) records which can be serialized to JSON. The schema is not known ahead of time and may change from one job to the next. We need to support browsing, querying and downloading the stored data. This was very easy to implement using MongoDB (easier than the alternatives available a few years ago) and it worked well for some time.

Posted by dotCore  |   Submitted: May 16 2013 / 02:05

Add your comment


Html tags not supported. Reply is editable for 5 minutes. Use [code lang="java|ruby|sql|css|xml"][/code] to post code snippets.

Recommended Links

Scala
Written by: Ryan Knight
Featured Refcardz: Top Refcardz:
  1. Apache Hadoop
  2. Play
  3. Akka
  4. Debugging JavaScript
  5. Design Patterns
  1. Apache Hadoop
  2. REST
  3. Java
  4. Git
  5. Java Performance
Connect with DZone