So I've been doing a goodreads scraping project for awhile. I'm looking to compare notes with other goodreads scraper folks. the below numbers give a flavor of what I've gotten so far. I just recently discovered the site map - so seems like there are 9.5 million author_id and 150,000 lists. The short of it is I have 18m book_id 400k detailed books (every work with more than 1000 ratings and more) 1.1m author_id 5.7m work_id 4087 distinct book tags 75k lists Books ---------------------------------------------------------------------- … books: field totals… … books: distinct counts… total : 17,987,973 100.0% ✓ complete author_id : 17,938,309 99.7% (49,664 missing) · 1,333,260 distinct ratings : 17,987,973 100.0% ✓ complete avg_rating : 17,978,021 99.9% (9,952 missing) · 415 distinct published : 4,081,236 22.7% (13,906,737 missing) pages : 381,066 2.1% (17,606,907 missing) · 1,920 distinct series_pos : 1,796,442 10.0% (16,191,531 missing) · 7,189 distinct genres : 360,926 2.0% (17,627,047 missing) tags : 753,422 4.2% (17,234,551 missing) work_id : 6,372,169 35.4% (11,615,804 missing) · 5,790,484 distinct ratings_zero : 11,563,215 · books with exactly 0 ratings … books: scanning genres/tags JSON (can take a while)… genres : distinct genre labels 1,410 tags : distinct tag/shelf names 4,079 Book Details total : 394,895 100.0% ✓ complete publisher : 389,552 98.6% (5,343 missing) · 52,603 distinct isbn13 : 278,329 70.5% (116,566 missing) isbn10 : 270,280 68.4% (124,615 missing) asin : 388,574 98.4% (6,321 missing) format : 393,267 99.6% (1,628 missing) · 346 distinct language : 385,411 97.6% (9,484 missing) · 97 distinct description : 390,975 99.0% (3,920 missing) series : 204,121 51.7% (190,774 missing) reviews_count : 394,610 99.9% (285 missing) ratings_dist : 394,610 99.9% (285 missing) currently_reading : 392,966 99.5% (1,929 missing) to_read : 394,203 99.8% (692 missing) editions_count : 368,108 93.2% (26,787 missing) Authors total : 1,084,031 100.0% ✓ complete average_rating : 792,753 73.1% (291,278 missing) num_ratings : 730,873 67.4% (353,158 missing) num_reviews : 621,310 57.3% (462,721 missing) num_shelves : 768,262 70.9% (315,769 missing) slug : 930,956 85.9% (153,075 missing) · 930,956 distinct catalog_pages : 847,609 78.2% (236,422 missing) source : 144,435 13.3% (939,596 missing) · 10 distinct Book Tags total rows : 4,572,460 distinct tags : 4,087 distinct book IDs : 781,592 distinct harvests : 6,937 position : 4,572,357 100.0% (103 missing) shelved : 4,572,460 100.0% ✓ complete shelved min : 0 max 3,165,909 avg 210.6   submitted by   /u/mjfmjfmjf [link]   [comments]