Constructed a server for handling MapReduce jobs, using Python, Git, WSL. Designed ‘Master’ to manage and distribute partitioned jobs via TCP; ‘Workers’ perform delegated tasks. Multi-processed & multi-threaded to parallelize work, and addressed busy-wait bottleneck via thread sleeps. Implemented fault-tolerance via UDP ‘heartbeats’ to ensure task completion.
abrahaja
authored
Name | Last commit | Last update |
---|---|---|
bin | ||
mapreduce | ||
tests | ||
README.md | ||
example_tcp_client.py | ||
example_tcp_server.py | ||
example_thread.py | ||
requirements.txt | ||
setup.py |