Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
E
eecs398-search
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Wiki
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Build
Pipelines
Jobs
Pipeline schedules
Artifacts
Deploy
Releases
Model registry
Operate
Environments
Monitor
Incidents
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
vcday
eecs398-search
Commits
fe658a59
Commit
fe658a59
authored
7 years ago
by
aanvi
Browse files
Options
Downloads
Patches
Plain Diff
Integrating wordData struct
parent
8daddacb
No related branches found
Branches containing commit
No related tags found
1 merge request
!2
WIP:Crawler parser 2 merge into duplicate url-crawler
Changes
3
Hide whitespace changes
Inline
Side-by-side
Showing
3 changed files
cmake-build-debug/CMakeFiles/clion-log.txt
+1
-4
1 addition, 4 deletions
cmake-build-debug/CMakeFiles/clion-log.txt
parser/Parser.h
+2
-1
2 additions, 1 deletion
parser/Parser.h
util/Tokenizer.h
+7
-5
7 additions, 5 deletions
util/Tokenizer.h
with
10 additions
and
10 deletions
cmake-build-debug/CMakeFiles/clion-log.txt
+
1
−
4
View file @
fe658a59
/Applications/CLion.app/Contents/bin/cmake/bin/cmake -DCMAKE_BUILD_TYPE=Debug -G "CodeBlocks - Unix Makefiles" /Users/veronicaday/Desktop/EECS398/eecs_398/project/eecs398-search
Toolchains are not configured Configure
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/veronicaday/Desktop/EECS398/eecs_398/project/eecs398-search/cmake-build-debug
This diff is collapsed.
Click to expand it.
parser/Parser.h
+
2
−
1
View file @
fe658a59
...
@@ -22,6 +22,7 @@ class Parser
...
@@ -22,6 +22,7 @@ class Parser
public:
public:
Parser
(
ProducerConsumerQueue
<
string
>
*
urlFrontierIn
)
Parser
(
ProducerConsumerQueue
<
string
>
*
urlFrontierIn
)
{
{
urlFrontier
=
urlFrontierIn
;
urlFrontier
=
urlFrontierIn
;
...
@@ -33,7 +34,7 @@ public:
...
@@ -33,7 +34,7 @@ public:
* @return
* @return
*/
*/
// TODO need to change vector type to word data, change where struct is declared
// TODO need to change vector type to word data, change where struct is declared
const
unordered_map
<
string
,
vector
<
int
>>
*
execute
(
Document
*
document
)
const
unordered_map
<
string
,
vector
<
Tokenizer
::
wordData
>>
*
execute
(
Document
*
document
)
{
{
Tokenizer
tokenizer
;
Tokenizer
tokenizer
;
parse
(
document
->
DocToString
(),
&
tokenizer
);
parse
(
document
->
DocToString
(),
&
tokenizer
);
...
...
This diff is collapsed.
Click to expand it.
util/Tokenizer.h
+
7
−
5
View file @
fe658a59
...
@@ -5,18 +5,20 @@
...
@@ -5,18 +5,20 @@
#include
<vector>
#include
<vector>
#include
"stringProcessing.h"
#include
"stringProcessing.h"
#include
"Stemmer.h"
#include
"Stemmer.h"
#include
"../parser/Parser.h"
using
namespace
std
;
using
namespace
std
;
struct
wordData
{
int
offset
;
int
frequency
=
0
;
//total num words/unique words??
};
class
Tokenizer
class
Tokenizer
{
{
public:
public:
struct
wordData
{
int
frequency
=
0
;
int
offset
;
};
Tokenizer
(
)
Tokenizer
(
)
{
{
docIndex
=
new
unordered_map
<
string
,
vector
<
wordData
>>
;
docIndex
=
new
unordered_map
<
string
,
vector
<
wordData
>>
;
...
...
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment