Commits · 93b118f30cb7ba56dd4d3361e214765271f1a6a6 · vcday / eecs398-search

Apr 05, 2018
- refactored queues and added timeouts · daa3194e
  benbergk authored 6 years ago
  
  daa3194e
- updated word tests indxer debugging · e95e6dd5
  jsclose authored 6 years ago
  
  e95e6dd5
- most up to date crawler/indexer · faebcefc
  jsclose authored 6 years ago
  
  faebcefc
Apr 04, 2018
- crawler can restart from a saved Queue · 9e71946f
  jsclose authored 6 years ago
  
  9e71946f
- Removing dead code, adding comments · 3408dee6
  jsclose authored 6 years ago
  
  3408dee6
- crawler now passing indexer the anchor text dict · 0f11d91c
  jsclose authored 6 years ago
  
  0f11d91c
Apr 03, 2018
- working with nick on end doc · 4ff7503e
  jsclose authored 6 years ago
  
  4ff7503e
- set up architecture for saving the anchor text after build is complete · 5ebf22d8
  jsclose authored 6 years ago
  
  5ebf22d8
Apr 02, 2018

building out more error handling for crawler, also can build for a certain... · 64ce4c5c

jsclose authored 6 years ago

building out more error handling for crawler, also can build for a certain number of docs mode, modified url crawling scoring

64ce4c5c

Mar 29, 2018
- modified local reader to take in a parsed url pointer to fix test case · 421e47ea
  jsclose authored 6 years ago
  
  421e47ea
- shutdown mechanism for the crawler + indexer with atomic bool working · 9a1995c4
  jsclose authored 6 years ago
  
  9a1995c4
- modifying queue rate · 24f56373
  jsclose authored 6 years ago
  
  24f56373
- housekeeping thread to write urls in queue to disk · 70e42436
  jsclose authored 6 years ago
  
  70e42436
- working tests, trying to add a shutdown method · 7b25b094
  jsclose authored 6 years ago
  
  7b25b094
Mar 28, 2018
- refactor · a63621a6
  jsclose authored 6 years ago
  
  a63621a6
Mar 27, 2018
- same as above · 43d67dc8
  jsclose authored 6 years ago
  
  43d67dc8
Mar 21, 2018
- hacked into making queries · c2667b49
  Nicholas Yang authored 6 years ago
  
  c2667b49
- removed the on disk doc map look upstuff, and created an isolated crawler test · 3cc58b8e
  jsclose authored 6 years ago
  
  3cc58b8e
Mar 20, 2018
- Style changes · 9ea59286
  vcday authored 6 years ago
  
  9ea59286
- indexer thread is now receiving from the parser · 36fc45a2
  jsclose authored 6 years ago
  
  36fc45a2
- intergrated indexer producer consumer queue · d8cc4e0c
  jsclose authored 6 years ago
  
  d8cc4e0c
- added a kill all spiders function so that we can start to terminate a run and... · 1e92c676
  jsclose authored 6 years ago
  
  added a kill all spiders function so that we can start to terminate a run and then restart it at the same point
  1e92c676
- Created a checkstatus function for the web readers so that we dont pull from a site that is bad · 4e2d4d5e
  jsclose authored 6 years ago
  
  4e2d4d5e
- modifiying duplicate url · 8364f3db
  jsclose authored 6 years ago
  
  8364f3db
- fixed bug (multiple initialization of ssl library) · 90ac15e7
  benbergk authored 6 years ago
  
  90ac15e7
Mar 19, 2018
- converted url frontier · b181bd3f
  jsclose authored 6 years ago
  
  b181bd3f
Mar 15, 2018
- fixing crawl/parse · f3a234af
  vcday authored 7 years ago
  
  f3a234af
Feb 18, 2018
- working OpenSSL · 81d48785
  jsclose authored 7 years ago
  
  81d48785
- working on ssl · 8f7de9b7
  jsclose authored 7 years ago
  
  8f7de9b7
Feb 15, 2018
- added command line args · e8774451
  benbergk authored 7 years ago
  
  e8774451
- update · 2f874e38
  jsclose authored 7 years ago
  
  2f874e38
Feb 14, 2018
- trying with 3 spiders · 1a4a3149
  jsclose authored 7 years ago
  
  1a4a3149
- modified seed url set up, working pull from the web · 0efd4c02
  jsclose authored 7 years ago
  
  0efd4c02
- fixed linker error, made formatting changes · 2693bb0f
  benbergk authored 7 years ago
  
  2693bb0f
Feb 13, 2018
- working on url testing + docMap work · e4c9b47b
  jsclose authored 7 years ago
  
  e4c9b47b
Feb 12, 2018
- working on shared docMap and docMapLookup · 32fcdac0
  jsclose authored 7 years ago
  
  32fcdac0
- working on doc map, url obj · 1cddffb7
  jsclose authored 7 years ago
  
  1cddffb7
Feb 06, 2018
- refactoring with utils · 0ae2c4d8
  jsclose authored 7 years ago
  
  0ae2c4d8
Feb 01, 2018
- Created Crawler Statistics · 3685c6b9
  benbergk authored 7 years ago
  
  3685c6b9
- Made thread class, fixed bugs · e6bbbee1
  benbergk authored 7 years ago
  
  e6bbbee1