improvements to RinetData class for processing csv files.

RinetData has a class variable that holds the names of the needed CSV files:

  @@csv_files = %w{students staff courses enrollments staff_assignments staff_sakai student_sakai}

RinetData be used from script/console or script/runner if you need to run the command from a shell.

Here's an example:

[rites (master)]$ jruby script/runner "rd = RinetData.new; rd.get_csv_files"
running Admin::Project.create_or_update_default_project_from_settings_yml

downloading: 07/students.csv and saving to: 
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/07/20091015_011519/students.csv
downloading: 07/staff.csv and saving to: 
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/07/20091015_011519/staff.csv
downloading: 07/courses.csv and saving to: 
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/07/20091015_011519/courses.csv
downloading: 07/enrollments.csv and saving to: 
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/07/20091015_011519/enrollments.csv
downloading: 07/staff_assignments.csv and saving to: 
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/07/20091015_011519/staff_assignments.csv
downloading: 07/staff_sakai.csv and saving to: 
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/07/20091015_011519/staff_sakai.csv
downloading: 07/student_sakai.csv and saving to: 
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/07/20091015_011519/student_sakai.csv

downloading: 16/students.csv and saving to: 
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/16/20091015_011519/students.csv
downloading: 16/staff.csv and saving to: 
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/16/20091015_011519/staff.csv
downloading: 16/courses.csv and saving to: 
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/16/20091015_011519/courses.csv
downloading: 16/enrollments.csv and saving to: 
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/16/20091015_011519/enrollments.csv
downloading: 16/staff_assignments.csv and saving to: 
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/16/20091015_011519/staff_assignments.csv
downloading: 16/staff_sakai.csv and saving to: 
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/16/20091015_011519/staff_sakai.csv
downloading: 16/student_sakai.csv and saving to: 
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/16/20091015_011519/student_sakai.csv

Here's an example of running the parser:

[rites (master)]$ jruby script/runner "rd = RinetData.new; rd.parse_csv_files"
running Admin::Project.create_or_update_default_project_from_settings_yml

parsing: 5328 rows from:
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/07/current/students.csv
parsing: 655 rows from:
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/07/current/staff.csv
parsing: 4115 rows from:
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/07/current/courses.csv
parsing: 52541 rows from:
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/07/current/enrollments.csv
parsing: 4329 rows from:
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/07/current/staff_assignments.csv
parsing: 928 rows from:
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/07/current/staff_sakai.csv
parsing: 8533 rows from:
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/07/current/student_sakai.csv

parsing: 1733 rows from:
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/16/current/students.csv
parsing: 207 rows from:
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/16/current/staff.csv
parsing: 961 rows from:
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/16/current/courses.csv
parsing: 16870 rows from:
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/16/current/enrollments.csv
parsing: 849 rows from:
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/16/current/staff_assignments.csv
parsing: 186 rows from:
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/16/current/staff_sakai.csv
parsing: 2610 rows from:
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/16/current/student_sakai.csv

The csv files are left in time-stamped directories rooted like this:

[rites (master)]$ ls -l rinet_data/districts/csv/07/
total 8
drwxrwxrwx   8 stephen  staff  272 Oct 15 00:50 20091015_004944
drwxrwxrwx  13 stephen  staff  442 Oct 15 01:15 20091015_005334
drwxrwxrwx   9 stephen  staff  306 Oct 15 01:04 20091015_010418
drwxrwxrwx   9 stephen  staff  306 Oct 15 01:12 20091015_011232
drwxrwxrwx   9 stephen  staff  306 Oct 15 01:13 20091015_011325
drwxrwxrwx   9 stephen  staff  306 Oct 15 01:15 20091015_011519
lrwxrwxrwx   1 stephen  staff   73 Oct 15 00:53 current -> /Users/stephen/dev/test/rites/rinet_data/districts/csv/07/20091015_005334
improvements to RinetData class for processing csv files.

RinetData has a class variable that holds the names of the needed CSV files:

  @@csv_files = %w{students staff courses enrollments staff_assignments staff_sakai student_sakai}

RinetData be used from script/console or script/runner if you need to run the command from a shell.

Here's an example:

[rites (master)]$ jruby script/runner "rd = RinetData.new; rd.get_csv_files"
running Admin::Project.create_or_update_default_project_from_settings_yml

downloading: 07/students.csv and saving to: 
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/07/20091015_011519/students.csv
downloading: 07/staff.csv and saving to: 
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/07/20091015_011519/staff.csv
downloading: 07/courses.csv and saving to: 
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/07/20091015_011519/courses.csv
downloading: 07/enrollments.csv and saving to: 
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/07/20091015_011519/enrollments.csv
downloading: 07/staff_assignments.csv and saving to: 
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/07/20091015_011519/staff_assignments.csv
downloading: 07/staff_sakai.csv and saving to: 
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/07/20091015_011519/staff_sakai.csv
downloading: 07/student_sakai.csv and saving to: 
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/07/20091015_011519/student_sakai.csv

downloading: 16/students.csv and saving to: 
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/16/20091015_011519/students.csv
downloading: 16/staff.csv and saving to: 
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/16/20091015_011519/staff.csv
downloading: 16/courses.csv and saving to: 
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/16/20091015_011519/courses.csv
downloading: 16/enrollments.csv and saving to: 
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/16/20091015_011519/enrollments.csv
downloading: 16/staff_assignments.csv and saving to: 
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/16/20091015_011519/staff_assignments.csv
downloading: 16/staff_sakai.csv and saving to: 
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/16/20091015_011519/staff_sakai.csv
downloading: 16/student_sakai.csv and saving to: 
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/16/20091015_011519/student_sakai.csv

Here's an example of running the parser:

[rites (master)]$ jruby script/runner "rd = RinetData.new; rd.parse_csv_files"
running Admin::Project.create_or_update_default_project_from_settings_yml

parsing: 5328 rows from:
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/07/current/students.csv
parsing: 655 rows from:
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/07/current/staff.csv
parsing: 4115 rows from:
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/07/current/courses.csv
parsing: 52541 rows from:
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/07/current/enrollments.csv
parsing: 4329 rows from:
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/07/current/staff_assignments.csv
parsing: 928 rows from:
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/07/current/staff_sakai.csv
parsing: 8533 rows from:
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/07/current/student_sakai.csv

parsing: 1733 rows from:
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/16/current/students.csv
parsing: 207 rows from:
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/16/current/staff.csv
parsing: 961 rows from:
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/16/current/courses.csv
parsing: 16870 rows from:
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/16/current/enrollments.csv
parsing: 849 rows from:
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/16/current/staff_assignments.csv
parsing: 186 rows from:
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/16/current/staff_sakai.csv
parsing: 2610 rows from:
  /Users/stephen/dev/test/rites/rinet_data/districts/csv/16/current/student_sakai.csv

The csv files are left in time-stamped directories rooted like this:

[rites (master)]$ ls -l rinet_data/districts/csv/07/
total 8
drwxrwxrwx   8 stephen  staff  272 Oct 15 00:50 20091015_004944
drwxrwxrwx  13 stephen  staff  442 Oct 15 01:15 20091015_005334
drwxrwxrwx   9 stephen  staff  306 Oct 15 01:04 20091015_010418
drwxrwxrwx   9 stephen  staff  306 Oct 15 01:12 20091015_011232
drwxrwxrwx   9 stephen  staff  306 Oct 15 01:13 20091015_011325
drwxrwxrwx   9 stephen  staff  306 Oct 15 01:15 20091015_011519
lrwxrwxrwx   1 stephen  staff   73 Oct 15 00:53 current -> /Users/stephen/dev/test/rites/rinet_data/districts/csv/07/20091015_005334

concord-consortium/rigse,concord-consortium/rigse,concord-consortium/rigse,concord-consortium/rigse,concord-consortium/rigse,concord-consortium/rigse