
Kamelot: OAR's scheduler in ocaml
=================================

Intro
-----
The main goal of this scheduler is to provide a better scalabily in comparaison to the schedulers in Perl.Up to now some features are missing see below.

This developement of this scheduler borrows lot of ideas and source codes from perl oar_2.x schedulers and (a large part of) moldable ocaml oar_1.6 scheduler (thanks respectively to Nicolas Capit and Lionel Eyraud for theirs codes). 

Features:
---------
 * conservative backfilling
 * resources properties matching
 * besteffort
 * hierarchies
 * multiple resource type [TO TEST]
 * multiple resource requests ( + ) [TO TEST]
 * time constant guards, [TO TEST]
 * suspend/resume, 
 * job depencies [TO TEST]
 * job container  
 * fairesharing [TO TEST]
 * order_by on resources [TO FINISH / EVALUATE]
 * ALL / BEST / BESTHALF for number of resources by level of hierarchy
 * moldable job
 * timesharing (alpha in separate version in kamelot_ts)
 * placeholder (alpha in separate version in kamelot_ts with timesharing)

Missing:
--------
 * Extensive test (no yet running on production cluster)
 * SCHEDULER_TOKEN_SCRIPTS support (for legacy licence management)
 * SCHEDULER_AVAILABLE_SUSPENDED_RESOURCE_TYPE (get_scheduled_jobs function is ready)

Depedencies:
------------
 * ocaml-core ocaml-libs ocaml-findlib libmysql-ocaml-dev libounit-ocaml-dev  libpostgresql-ocaml-dev


Next:
-----

 * SCHEDULER_TOKEN_SCRIPTS support (for legacy licence management)
 * Support for null ???
 * hierarchy automatic extratction doesn't scale, need to be precomputed (in lua ???)
   - need to serialize use of sudo apt-get install libatdgen-ocaml-dev
 * test hierarchy construction with different type of resource (exception raises when a field is missing)
 * performance testing
 * add always SCHEDULER_RESOURCES_ALWAYS_ASSIGNED_TYPE (is it really needed ?)

 * scheduler message (see perl version )
 * job_error / job_message / scheduler message
 * need to test multi-resource-type (since >= cbf_mb_h)
 * need to test multi-request with non exclusive resource selection (since >= cbf_mb_h)
 * errors logging (at least same error support as provide in perl scheduler)
 * nb_asked_resource = 0 raise an error (>= cbf_mb_h) 
 * unit test
 * better compilation process (for unit tests)

ToDo:
-----

  * debug noschedule job issue
  * test timesharing
  * test placeholder

  * test_unit: better compilation process
  * Ounit (cf archive)
    * test sub_intevals

Misc:
-----

  * With 64 bits machine we can use ocaml's int with 63 bits instead of Int64.
  * For json serialization  use biniou/atdegen  (sudo apt-get install libatdgen-ocaml-dev)

Done:
-----
 * moldable job support
 * resource order_by support (usable) 
 * container
 * Support of postgresql
 * Preliminary performance comparaison (perl version timesharing only scheduler from oar-server_2.3.4-1_all.deb against cbf_mh_h). Perl scheduler doesn't seem to scale with number of resources)
 * modify itv_intersect in Interval / remove itv2str, itvs2str (>= cbf_mh_h)
 * multi-resource-type (since >= cbf_mh_h) (
 * multi-request with non exclusive resource selection (since >= cbf_mh_h)

Remarks and misc:
-----------------
 * http://martin.jambon.free.fr/ocaml.htm 

Bugs:
-----
 * BestEffort after a while remain at waiting state
 * test scalable if with available_upto enable with very large job (> 10K resources)

Debug:
------
  make bc
  ocamlmktop -I /usr/lib/ocaml/  -o yop str.cma unix.cma ./common/helpers.cmo ./common/interval.cmo ./common/conf.cmo ./common/types.cmo ./common/hierarchy.cmo ./scheduling.cmo

ocamlmktop -I /usr/lib/ocaml/  -o yop str.cma unix.cma ./common/helpers.cmo \
            ./common/interval.cmo ./common/conf.cmo ./common/types.cmo ./common/hierarchy.cmo \
            ./scheduling.cmo mysql/mysql.cma ./common/mysql_driver.cmo iolib.cmo ./kamelot.cmo

ocamlmktop -I /usr/lib/ocaml/  -o yop str.cma unix.cma ./common/helpers.cmo \
            ./common/interval.cmo ./common/conf.cmo ./common/types.cmo ./common/hierarchy.cmo \
            ./scheduling.cmo mysql/mysql.cma ./common/mysql_driver.cmo iolib.cmo 

ocamlmktop -I /usr/lib/ocaml/  -o yop str.cma unix.cma ./common/helpers.cmo \
            ./common/interval.cmo ./common/conf.cmo ./common/types_ts.cmo ./common/hierarchy.cmo \
            ./scheduling_ts.cmo mysql/mysql.cma ./common/mysql_driver.cmo iolib_ts.cmo 



make POSTGRESQL=true clean bc

ocamlmktop -I /usr/lib/ocaml/  -o yop str.cma unix.cma bigarray.cma threads/threads.cma ./common/helpers.cmo \
            ./common/interval.cmo ./common/conf.cmo ./common/types.cmo ./common/hierarchy.cmo \
            ./scheduling.cmo postgresql/postgresql.cma ./common/postgresql_driver.cmo iolib.cmo



rlwrap ./yop -I ./common -I .
