Fork me on GitHub

JPregel





Source Code

GitHub lets you download the source in multiple formats





You can also clone the project with Git by running:

$ git clone git://github.com/kowshik/JPregel



How-to build JPregel system and dev packages :

  1. Download the source code and extract the archive.
  2. Modify build.xml to configure hosts in your compute cluster. Setup a headless user account for SSHing in your cluster, and update the same in the build.xml.
  3. Run the following command to compile :

    $ ant compile
  4. Run the following command to build JPregel. The JAR file : jpregel.jar is created in dist/.

    $ ant build_jpregel
  5. Run the following command to create JPregel Dev Package. The JAR file : jpregel_dev.jar is created in dist/.

    $ ant build_dev

How-to create your JPregel application and deploy the system :

  1. Extend class api.Vertex to create your own Vertex class. Lets call it ClientVertex.java.
  2. Compile ClientVertex.java using the command :

    $ javac -cp ClientVertex.java
  3. Build application.jar using the command :

    $ jar cvf application.jar

    Example : jar cvf application ClientVertex.class
  4. Copy your input graph file (adjacency list representation) to graph.dat under the same folder. Alternatively, you can also generate random graphs using the command :

    $ ant runGraphGenerator -Dmax_cost=<max_cost_value> -Dnum_vertices=<num_vertices_in_graph>

    Example : ant runGraphGenerator -Dmax_cost=1000 -Dnum_vertices=25, generates graph.dat file containing adjacency list of a directed graph of 25 vertices with cost of each outgoing edge from a vertex lying between 1 and 1000.
  5. Run a remote JPregel master using the command :

    $ ant runRemoteMaster -Dusername=<SSH_username> -Dapp=<absolute_path_to_application.jar> -Dvertex_class=<your_vertex_class> -Dnum_machines=<num_worker_machines>

    Example : ant runRemoteMaster -Dusername=blahblah -Dapp=/home/blahblah/application.jar -Dvertex_class=ClientVertex -Dnum_machines=16
  6. Run remote JPregel workers using the command (XX refers to number of worker managers. See "ant-projecthelp"):

    $ ant runXXWkrManagers -Dusername=<SSH_username> -Dapp=<absolute_path_to_application.jar> -Dvertex_class=<your_vertex_class>

    Example : ant run16WkrManagers -Dusername=blahblah -Dapp=/home/blahblah/application.jar -Dvertex_class=ClientVertex
  7. Once the specified machines complete registration with the master, the execution of your application will commence. Logs will be written to the folder logs/. Checkpointing data is written to checkpoints/. Solutions is written to solutions/ upon completion of computations.









© 2010 Kowshik Prakasam and Manasa Chandrasekhar | Template design by Andreas Viklund