Commits · 815d04fd04862aed4f66257c68d72ca68e7b4592 · vcday / eecs398-search

Apr 02, 2018
- washington post build, crawler gets 1600 docs, all from same site · 815d04fd
  jsclose authored 7 years ago
  
  815d04fd
- building out more error handling for crawler, also can build for a certain... · 64ce4c5c
  jsclose authored 7 years ago
  64ce4c5c
Mar 30, 2018
- url->anchortext map working, added addiontal features for checking valid url · fc704f06
  jsclose authored 7 years ago
  
  fc704f06
Mar 29, 2018
- url parser error handling · 01c790b6
  jsclose authored 7 years ago
  
  01c790b6
- modified local reader to take in a parsed url pointer to fix test case · 421e47ea
  jsclose authored 7 years ago
  
  421e47ea
- shutdown mechanism for the crawler + indexer with atomic bool working · 9a1995c4
  jsclose authored 7 years ago
  
  9a1995c4
- modifying queue rate · 24f56373
  jsclose authored 7 years ago
  
  24f56373
- housekeeping thread to write urls in queue to disk · 70e42436
  jsclose authored 7 years ago
  
  70e42436
- working tests, trying to add a shutdown method · 7b25b094
  jsclose authored 7 years ago
  
  7b25b094
Mar 28, 2018
- more tests passing · 23693af7
  jsclose authored 7 years ago
  
  23693af7
- refactor · a63621a6
  jsclose authored 7 years ago
  
  a63621a6
Mar 27, 2018
- same as above · 43d67dc8
  jsclose authored 7 years ago
  
  43d67dc8
- modifiying all of the parsed urls to parsed url pointers for better error... · fe84ad69
  jsclose authored 7 years ago
  fe84ad69
Mar 22, 2018
- parser2test · 21ea7013
  vcday authored 7 years ago
  
  21ea7013
Mar 21, 2018
- made simple test for parser with new style · 1ab1c97d
  benbergk authored 7 years ago
  
  1ab1c97d
- changed ParsedUrl to be strings · 9d465238
  benbergk authored 7 years ago
  
  9d465238
- stable crawler · 3d5f1a5e
  jsclose authored 7 years ago
  
  3d5f1a5e
- error checking for readers and end of html check for parser · 8eaaf7c5
  jsclose authored 7 years ago
  
  8eaaf7c5
- testing integration · eee8ee6b
  jsclose authored 7 years ago
  
  eee8ee6b
- fixed stemming bug when it tried to stem just the letter s, caused a weird infinite loop · 6efe9a83
  jsclose authored 7 years ago
  
  6efe9a83
- removed the on disk doc map look upstuff, and created an isolated crawler test · 3cc58b8e
  jsclose authored 7 years ago
  
  3cc58b8e
- code reformat for style · 58f44a4f
  jsclose authored 7 years ago
  
  58f44a4f
- push · 9f3b7562
  jsclose authored 7 years ago
  
  9f3b7562
- fixed bug in SR_factory · d5831e07
  benbergk authored 7 years ago
  
  d5831e07
- added test url variable to LocalReader · 0cab7f22
  benbergk authored 7 years ago
  
  0cab7f22
- began fixing local reader · c1a883ba
  benbergk authored 7 years ago
  
  c1a883ba
Mar 20, 2018
- Style changes · 9ea59286
  vcday authored 7 years ago
  
  9ea59286
- modified point · ce4a804e
  jsclose authored 7 years ago
  
  ce4a804e
- merge conflict · 1cef72a4
  vcday authored 7 years ago
  
  1cef72a4
- indexer thread is now receiving from the parser · 36fc45a2
  jsclose authored 7 years ago
  
  36fc45a2
- intergrated indexer producer consumer queue · d8cc4e0c
  jsclose authored 7 years ago
  
  d8cc4e0c
- added a kill all spiders function so that we can start to terminate a run and... · 1e92c676
  jsclose authored 7 years ago
  1e92c676
- Created a checkstatus function for the web readers so that we dont pull from a site that is bad · 4e2d4d5e
  jsclose authored 7 years ago
  
  4e2d4d5e
- modifiying duplicate url · 8364f3db
  jsclose authored 7 years ago
  
  8364f3db
- fixed bug (multiple initialization of ssl library) · 90ac15e7
  benbergk authored 7 years ago
  
  90ac15e7
Mar 19, 2018
- crawler-parser test consistent · 02e3c897
  vcday authored 7 years ago
  
  02e3c897
- change string pointers to index · d95e47de
  vcday authored 7 years ago
  
  d95e47de
- fixed PageToString function · 7e222684
  benbergk authored 7 years ago
  
  7e222684
- added PageToString functions · 262975b9
  benbergk authored 7 years ago
  
  262975b9
- converted url frontier · b181bd3f
  jsclose authored 7 years ago
  
  b181bd3f