** make_installer.pl slowness **

Use -d:DProf & dprofpp:

---- installer::setupscript.pm::get_all_items_from_script ----

+ one of the worst parsers I've -ever- seen
	+ initially - it read lots of lines multiple times [!]
	+ the solution is:
		+ write a good parser instead [!]
		+ call it just once for the whole file.
		+ use simpler regexps

is it trailing / to the '$' regexps that are the problem ?

%Time ExclSec CumulS #Calls sec/call Csec/c  Name
 57.1   2.125  2.451      8   0.2657 0.3063  installer::setupscript::get_all_it
                                             ems_from_script
 8.50   0.316  0.316  54248   0.0000 0.0000  installer::remover::remove_leading
                                             _and_ending_quotationmarks
to: with some chomps ...
%Time ExclSec CumulS #Calls sec/call Csec/c  Name
 55.5   2.115  2.491      8   0.2644 0.3113  installer::setupscript::get_all_it
                                             ems_from_script
 9.61   0.366  0.366  54248   0.0000 0.0000  installer::remover::remove_leading
                                             _and_ending_quotationmarks

---- installer::setupscript::replace_all_setupscriptvariables_in_script ----

Sped this beasty up:

before:
 33.4   2.030  2.030      1   2.0300 2.0300  installer::setupscript::replace_al
                                             l_setupscriptvariables_in_script
to:
 6.43   0.240  0.240      1   0.2400 0.2400  installer::setupscript::replace_al
                                             l_setupscriptvariables_in_script

	just by switching to one big string.

---- installer::scriptitems::get_string_of_modulegids_for_itemgid ----

Before:

%Time ExclSec CumulS #Calls sec/call Csec/c  Name
 45.9   8.158  8.168   1025   0.0080 0.0080  installer::scriptitems::get_string_of_modulegids_for_itemgid


After: vanished ...

+ why is it so bad ?

The modulegids_for_itemgid problem is simple:
	+ we have lists of files in each module,
	+ but not lists of modules for each file
		+ we try to fix this.
	+ unfortunately the order of this is extremely high.
	+ surely better to:
		+ walk this backwards: foreach module
			+ split & append item name to each gid ...


--------

* 20 seconds ++

  1816320   0.74356  20.19000   337:   $newhash{$key} = ${$oldcollector}[$i]-

	+ modules/installer/converter.pm - convert_array_to_hash
		- 1.8 million calls [ 20 seconds ]

--------

* 30 seconds ++

  1215650   4.33675  20.69000  2383:  if ( $allitems =~ /\b$itemgid\b/i )

modules/installer/scriptitems.pm:sub get_string_of_modulegids_for_itemgid

     1025   0.00002   0.01000  2376: for ( my $i = 0; $i <= $#{$modulesref}; $i++
        0   0.00000   0.00000  2377: {
  1215650   0.07093  11.30000  2378:  my $onemodule = ${$modulesref}[$i];
  1215650   0.11160  10.78000  2379:  my $allitems = "";

--------

* 5 seconds ++

   503880   0.03416   4.62000   283:  my $line = ${$scriptref}[$i];
   503880   0.08359   5.76000   285:  if ( $line =~


--------

   148324   0.10664   1.57000   335:  foreach $key (keys %{${$oldcollector}[$i]})
   161800   0.02043   1.41000    42: my ( $stringref ) = @_;
   161800   0.03524   1.77000    44: $$stringref =~ s/^\s*//;
   161800   0.43650   2.52000    45: $$stringref =~ s/\s*$//;
   168455   0.00573   1.50000   872:  my $directorygid = $onedirectory->{'gid'};
   168455   0.01485   1.58000   871:  $onedirectory = ${$dirsarrayref}[$i];
   168455   0.02226   1.52000   874:  if ($directorygid eq $searchgid)
   179687   0.02017   1.79000   291:    my $packagemodule =
   179687   0.03661   1.62000   290:    if ( $includefile ) { next; }
   179687   0.08935   1.77000   293:    if ( $filemodule eq $packagemodule )
   180736   0.01710   1.85000    46:  if ( ${$arrayref}[$i] eq $searchstring)
   218740   0.01105   1.91000   263:    ${$scriptref}[$i] = $line;
   218740   0.01886   1.64000   261:    my $value = $subs{$key};
   218740   0.56364   3.96000   262:    $line =~ s/$key/$value/g;
   503880   0.03416   4.62000   283:  my $line = ${$scriptref}[$i];
   503880   0.08359   5.76000   285:  if ( $line =~
  1215650   0.07093  11.30000  2378:  my $onemodule = ${$modulesref}[$i];
  1215650   0.09240  11.14000  2381:  if ( $onemodule->{$itemname} ) { $allitems
  1215650   0.11160  10.78000  2379:  my $allitems = "";
  1215650   4.33675  20.69000  2383:  if ( $allitems =~ /\b$itemgid\b/i )
  1816320   0.74356  20.19000   337:   $newhash{$key} = ${$oldcollector}[$i]-


New:

... analyzing directories ... 
return 313 items for 'Directory'
... analyzing files ... 
return 981 items for 'File'
... analyzing scpactions ... 
return 34 items for 'ScpAction'
... analyzing shortcuts ... 
return 8 items for 'Shortcut'
... analyzing unix links ... 
return 2 items for 'Unixlink'
... analyzing profile ... 
return 12 items for 'Profile'
... analyzing profileitems ... 
return 105 items for 'ProfileItem'
... analyzing modules ... 
return 1198 items for 'Module'

before:
... analyzing directories ... 
return 313 items for 'Directory'
... analyzing files ... 
return 981 items for 'File'
... analyzing scpactions ... 
return 34 items for 'ScpAction'
... analyzing shortcuts ... 
return 8 items for 'Shortcut'
... analyzing unix links ... 
return 2 items for 'Unixlink'
... analyzing profile ... 
return 12 items for 'Profile'
... analyzing profileitems ... 
return 105 items for 'ProfileItem'
... analyzing modules ... 
return 1198 items for 'Module'


Total Elapsed Time = 14.82739 Seconds
  User+System Time = 6.077393 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c  Name
 35.3   2.149  2.298      8   0.2686 0.2872  installer::setupscript::get_all_it
                                             ems_from_script
 33.4   2.030  2.030      1   2.0300 2.0300  installer::setupscript::replace_al
                                             l_setupscriptvariables_in_script
 5.25   0.319  0.693     63   0.0051 0.0110  main::BEGIN
 4.23   0.257  0.257    482   0.0005 0.0005  installer::systemactions::read_com
                                             plete_directory
 3.95   0.240  0.240      2   0.1200 0.1200  installer::scpzipfiles::replace_al
                                             l_ziplistvariables_in_file
 2.65   0.161  0.161   2453   0.0001 0.0001  installer::existence::exists_in_ar
                                             ray
 2.30   0.140  0.140      5   0.0280 0.0280  installer::files::read_file
 2.14   0.130  0.130  54248   0.0000 0.0000  installer::remover::remove_leading
                                             _and_ending_quotationmarks
 1.15   0.070  0.070      1   0.0700 0.0700  installer::scriptitems::resolve_al
                                             l_directory_names
 0.99   0.060  0.257      5   0.0120 0.0514  installer::exiter::BEGIN
 0.81   0.049  0.049  11694   0.0000 0.0000  installer::scriptitems::insert_for
                                             _item
 0.66   0.040  0.197     17   0.0023 0.0116  installer::worker::BEGIN
 0.59   0.036  0.056   1007   0.0000 0.0001  installer::scriptitems::get_string


