Refactor RemoveDuplicates: use HashSet and List
Refactor RemoveDuplicates: use HashSet and List

The RemoveDuplicates task has been refactored to use `HashSet<string>` and
`List<ITaskItem>` from `System.Collections.Generic` over the previous
non-generic `Hashtable` and `ArrayList` implementation.

Recent versions of `HashSet<T>` [have constructors that take a
capacity][1], but these changes are not available in all the different
versions of .NET that MSBuild is built with.

I tested this implementation against the original
`Hashtable`/`Arraylist` implementation on different item collections.
This version looks to perform better or on par.

Aside from empty inputs, three different types of inputs of varying
lengths (10, 20, 100, and 100,000 items) were tested:

* Diff: "itemspec1", "itemspec2", ..., "itemspecn"
* Mix: "itemspec1", "itemspec2", ... with 95% unique and 5% duplicates
  of some other item, all shuffled together
* Same: "itemspec", "itemspec", ...

BenchmarkDotNet=v0.10.12, OS=Windows 10 Redstone 3 [1709, Fall Creators Update] (10.0.16299.248)
Intel Xeon CPU E5645 2.40GHz, 2 CPU, 24 logical cores and 12 physical cores
Frequency=2337889 Hz, Resolution=427.7363 ns, Timer=TSC
  [Host]       : .NET Framework 4.6.2 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.2633.0
  Clr          : .NET Framework 4.6.2 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.2633.0
  LegacyJitX86 : .NET Framework 4.6.2 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.2633.0
  RyuJitX86    : .NET Framework 4.6.2 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.2633.0

Platform=X86  Runtime=Clr

      Method |          Job |       Jit |   Method |      Mean |     Error |    StdDev |    Median |
------------ |------------- |---------- |--------- |----------:|----------:|----------:|----------:|
 Diff100_000 |          Clr | LegacyJit |      New | 30.395 ms | 0.1348 ms | 0.1195 ms | 30.422 ms |
 Diff100_000 |          Clr | LegacyJit |      Old | 44.874 ms | 0.3963 ms | 0.3707 ms | 44.742 ms |
 Diff100_000 | LegacyJitX86 | LegacyJit |      New | 30.308 ms | 0.1644 ms | 0.1458 ms | 30.313 ms |
 Diff100_000 | LegacyJitX86 | LegacyJit |      Old | 44.414 ms | 0.1064 ms | 0.0944 ms | 44.417 ms |
 Diff100_000 |    RyuJitX86 |    RyuJit |      New | 30.126 ms | 0.1119 ms | 0.0992 ms | 30.107 ms |
 Diff100_000 |    RyuJitX86 |    RyuJit |      Old | 45.206 ms | 0.4861 ms | 0.4547 ms | 45.068 ms |
     Diff100 |          Clr | LegacyJit |      New |  9.859 ms | 0.0365 ms | 0.0324 ms |  9.850 ms |
     Diff100 |          Clr | LegacyJit |      Old |  9.843 ms | 0.0199 ms | 0.0155 ms |  9.844 ms |
     Diff100 | LegacyJitX86 | LegacyJit |      New |  9.798 ms | 0.0343 ms | 0.0321 ms |  9.781 ms |
     Diff100 | LegacyJitX86 | LegacyJit |      Old |  9.788 ms | 0.0235 ms | 0.0208 ms |  9.791 ms |
     Diff100 |    RyuJitX86 |    RyuJit |      New |  9.599 ms | 0.0247 ms | 0.0219 ms |  9.594 ms |
     Diff100 |    RyuJitX86 |    RyuJit |      Old |  9.859 ms | 0.0668 ms | 0.0625 ms |  9.828 ms |
      Diff10 |          Clr | LegacyJit |      New |  9.685 ms | 0.0201 ms | 0.0168 ms |  9.683 ms |
      Diff10 |          Clr | LegacyJit |      Old |  9.590 ms | 0.0212 ms | 0.0165 ms |  9.583 ms |
      Diff10 | LegacyJitX86 | LegacyJit |      New |  9.906 ms | 0.0169 ms | 0.0150 ms |  9.903 ms |
      Diff10 | LegacyJitX86 | LegacyJit |      Old |  9.704 ms | 0.0346 ms | 0.0307 ms |  9.695 ms |
      Diff10 |    RyuJitX86 |    RyuJit |      New |  9.720 ms | 0.0151 ms | 0.0134 ms |  9.719 ms |
      Diff10 |    RyuJitX86 |    RyuJit |      Old |  9.845 ms | 0.0492 ms | 0.0436 ms |  9.843 ms |
      Diff20 |          Clr | LegacyJit |      New |  9.742 ms | 0.0850 ms | 0.0754 ms |  9.717 ms |
      Diff20 |          Clr | LegacyJit |      Old |  9.792 ms | 0.0736 ms | 0.0688 ms |  9.770 ms |
      Diff20 | LegacyJitX86 | LegacyJit |      New |  9.827 ms | 0.0244 ms | 0.0216 ms |  9.827 ms |
      Diff20 | LegacyJitX86 | LegacyJit |      Old |  9.877 ms | 0.0240 ms | 0.0187 ms |  9.879 ms |
      Diff20 |    RyuJitX86 |    RyuJit |      New |  9.790 ms | 0.0265 ms | 0.0222 ms |  9.784 ms |
      Diff20 |    RyuJitX86 |    RyuJit |      Old |  9.850 ms | 0.0512 ms | 0.0454 ms |  9.838 ms |
       Empty |          Clr | LegacyJit |      New |  9.746 ms | 0.0083 ms | 0.0078 ms |  9.747 ms |
       Empty |          Clr | LegacyJit |      Old |  9.861 ms | 0.0893 ms | 0.0836 ms |  9.830 ms |
       Empty | LegacyJitX86 | LegacyJit |      New |  9.656 ms | 0.0138 ms | 0.0116 ms |  9.655 ms |
       Empty | LegacyJitX86 | LegacyJit |      Old |  9.709 ms | 0.0916 ms | 0.0765 ms |  9.681 ms |
       Empty |    RyuJitX86 |    RyuJit |      New |  9.675 ms | 0.0309 ms | 0.0274 ms |  9.666 ms |
       Empty |    RyuJitX86 |    RyuJit |      Old |  9.823 ms | 0.0174 ms | 0.0163 ms |  9.819 ms |
  Mix100_000 |          Clr | LegacyJit |      New | 42.243 ms | 0.2917 ms | 0.2585 ms | 42.347 ms |
  Mix100_000 |          Clr | LegacyJit |      Old | 58.505 ms | 0.4211 ms | 0.3733 ms | 58.470 ms |
  Mix100_000 | LegacyJitX86 | LegacyJit |      New | 42.255 ms | 0.2787 ms | 0.2607 ms | 42.269 ms |
  Mix100_000 | LegacyJitX86 | LegacyJit |      Old | 57.235 ms | 0.2927 ms | 0.2444 ms | 57.222 ms |
  Mix100_000 |    RyuJitX86 |    RyuJit |      New | 41.249 ms | 0.5285 ms | 0.4685 ms | 41.126 ms |
  Mix100_000 |    RyuJitX86 |    RyuJit |      Old | 57.771 ms | 0.5596 ms | 0.5235 ms | 57.726 ms |
      Mix100 |          Clr | LegacyJit |      New |  9.798 ms | 0.0245 ms | 0.0177 ms |  9.797 ms |
      Mix100 |          Clr | LegacyJit |      Old |  9.881 ms | 0.0212 ms | 0.0188 ms |  9.887 ms |
      Mix100 | LegacyJitX86 | LegacyJit |      New |  9.859 ms | 0.0388 ms | 0.0344 ms |  9.852 ms |
      Mix100 | LegacyJitX86 | LegacyJit |      Old |  9.657 ms | 0.0174 ms | 0.0136 ms |  9.654 ms |
      Mix100 |    RyuJitX86 |    RyuJit |      New |  9.847 ms | 0.0150 ms | 0.0133 ms |  9.848 ms |
      Mix100 |    RyuJitX86 |    RyuJit |      Old |  9.750 ms | 0.0683 ms | 0.0570 ms |  9.749 ms |
       Mix10 |          Clr | LegacyJit |      New |  9.822 ms | 0.0198 ms | 0.0165 ms |  9.823 ms |
       Mix10 |          Clr | LegacyJit |      Old |  9.683 ms | 0.0342 ms | 0.0303 ms |  9.674 ms |
       Mix10 | LegacyJitX86 | LegacyJit |      New |  9.574 ms | 0.0514 ms | 0.0456 ms |  9.564 ms |
       Mix10 | LegacyJitX86 | LegacyJit |      Old |  9.819 ms | 0.0263 ms | 0.0190 ms |  9.820 ms |
       Mix10 |    RyuJitX86 |    RyuJit |      New |  9.683 ms | 0.0766 ms | 0.0679 ms |  9.652 ms |
       Mix10 |    RyuJitX86 |    RyuJit |      Old |  9.856 ms | 0.1947 ms | 0.1821 ms |  9.821 ms |
       Mix20 |          Clr | LegacyJit |      New |  9.950 ms | 0.0467 ms | 0.0414 ms |  9.937 ms |
       Mix20 |          Clr | LegacyJit |      Old |  9.899 ms | 0.0255 ms | 0.0226 ms |  9.892 ms |
       Mix20 | LegacyJitX86 | LegacyJit |      New |  9.891 ms | 0.0239 ms | 0.0200 ms |  9.891 ms |
       Mix20 | LegacyJitX86 | LegacyJit |      Old |  9.625 ms | 0.0274 ms | 0.0243 ms |  9.624 ms |
       Mix20 |    RyuJitX86 |    RyuJit |      New |  9.837 ms | 0.0094 ms | 0.0083 ms |  9.837 ms |
       Mix20 |    RyuJitX86 |    RyuJit |      Old |  9.867 ms | 0.0355 ms | 0.0314 ms |  9.868 ms |
 Same100_000 |          Clr | LegacyJit |      New | 10.754 ms | 0.0574 ms | 0.0537 ms | 10.739 ms |
 Same100_000 |          Clr | LegacyJit |      Old | 10.651 ms | 0.0208 ms | 0.0163 ms | 10.645 ms |
 Same100_000 | LegacyJitX86 | LegacyJit |      New | 10.646 ms | 0.1451 ms | 0.1357 ms | 10.591 ms |
 Same100_000 | LegacyJitX86 | LegacyJit |      Old | 10.763 ms | 0.0222 ms | 0.0197 ms | 10.762 ms |
 Same100_000 |    RyuJitX86 |    RyuJit |      New | 10.713 ms | 0.0241 ms | 0.0213 ms | 10.709 ms |
 Same100_000 |    RyuJitX86 |    RyuJit |      Old | 10.681 ms | 0.0693 ms | 0.0614 ms | 10.657 ms |
     Same100 |          Clr | LegacyJit |      New |  9.692 ms | 0.0155 ms | 0.0130 ms |  9.688 ms |
     Same100 |          Clr | LegacyJit |      Old |  9.843 ms | 0.0256 ms | 0.0227 ms |  9.838 ms |
     Same100 | LegacyJitX86 | LegacyJit |      New |  9.985 ms | 0.0180 ms | 0.0150 ms |  9.980 ms |
     Same100 | LegacyJitX86 | LegacyJit |      Old |  9.905 ms | 0.0608 ms | 0.0539 ms |  9.883 ms |
     Same100 |    RyuJitX86 |    RyuJit |      New |  9.618 ms | 0.0165 ms | 0.0137 ms |  9.615 ms |
     Same100 |    RyuJitX86 |    RyuJit |      Old |  9.761 ms | 0.0936 ms | 0.0830 ms |  9.722 ms |
      Same10 |          Clr | LegacyJit |      New |  9.672 ms | 0.0379 ms | 0.0317 ms |  9.664 ms |
      Same10 |          Clr | LegacyJit |      Old |  9.830 ms | 0.0824 ms | 0.0731 ms |  9.809 ms |
      Same10 | LegacyJitX86 | LegacyJit |      New |  9.875 ms | 0.0490 ms | 0.0434 ms |  9.854 ms |
      Same10 | LegacyJitX86 | LegacyJit |      Old |  9.736 ms | 0.0574 ms | 0.0509 ms |  9.724 ms |
      Same10 |    RyuJitX86 |    RyuJit |      New |  9.834 ms | 0.0408 ms | 0.0361 ms |  9.821 ms |
      Same10 |    RyuJitX86 |    RyuJit |      Old |  9.735 ms | 0.0265 ms | 0.0235 ms |  9.741 ms |
      Same20 |          Clr | LegacyJit |      New |  9.793 ms | 0.0715 ms | 0.0668 ms |  9.759 ms |
      Same20 |          Clr | LegacyJit |      Old |  9.819 ms | 0.0319 ms | 0.0299 ms |  9.823 ms |
      Same20 | LegacyJitX86 | LegacyJit |      New |  9.846 ms | 0.0270 ms | 0.0239 ms |  9.838 ms |
      Same20 | LegacyJitX86 | LegacyJit |      Old |  9.869 ms | 0.0330 ms | 0.0293 ms |  9.864 ms |
      Same20 |    RyuJitX86 |    RyuJit |      New |  9.784 ms | 0.0173 ms | 0.0153 ms |  9.779 ms |
      Same20 |    RyuJitX86 |    RyuJit |      Old |  9.632 ms | 0.0301 ms | 0.0267 ms |  9.632 ms |

[1]: https://github.com/dotnet/corefx/commit/bc7cd2bca54441a83ecfea145ef59e936a17a18c

AndyGerlicher/msbuild,jeffkl/msbuild,rainersigwald/msbuild,AndyGerlicher/msbuild,Microsoft/msbuild,Microsoft/msbuild,jeffkl/msbuild,rainersigwald/msbuild,rainersigwald/msbuild,chipitsine/msbuild,chipitsine/msbuild,cdmihai/msbuild,jeffkl/msbuild,mono/msbuild,sean-gilliam/msbuild,AndyGerlicher/msbuild,Microsoft/msbuild,cdmihai/msbuild,AndyGerlicher/msbuild,sean-gilliam/msbuild,mono/msbuild,Microsoft/msbuild,chipitsine/msbuild,rainersigwald/msbuild,mono/msbuild,jeffkl/msbuild,cdmihai/msbuild,mono/msbuild,chipitsine/msbuild,sean-gilliam/msbuild,cdmihai/msbuild,sean-gilliam/msbuild