Tuesday, April 17, 2012

cabal bench

With the release of cabal-install 0.14, you can use Cabal to build and run your benchmarks. No more Makefiles that try to replicate things Cabal already does. Just like cabal test allows us to automatically find and run tests using e.g. a buildbot, this feature should let us track performance of libraries over time and catch performance regressions earlier. There's some future work needed to make this convenient, but it's already possible today if you're willing to parse the output of your favorite benchmarking library (e.g. Criterion.)

Writing benchmark sections in your Cabal file is very similar to writing test suite sections. Here's an example taken from the bytestring package:

benchmark bench-builder-all
  type:             exitcode-stdio-1.0
  hs-source-dirs:   . bench
  main-is:          BenchAll.hs
  build-depends:    base, deepseq, ghc-prim,
  c-sources:        cbits/fpstring.c
  include-dirs:     include
  ghc-options:      -O2

To run just do:

cabal configure --enable-benchmarks && cabal build && cabal bench


  1. That's quite cool ! But What's the requirements related to version ? Does the remaining package works if the benchmark section is not understood ?

    1. I believe Cabal as far back as 1.8 or 1.10 can parse unknown sections and ignore them (with a warning.) You should be able to stick a benchmarking section in your library even if your users haven't upgraded to the latest version of Cabal yet.

  2. Would it not be a good idea to enable the developer to make different targets?

    I mean, instead of having a testing and a benchmarking target, you could have a general syntax to declare a target, like:

    target bench bench-builder-all
    type: ....
    hs-source-dirs: ....

    target test debug-executable

    And then to activate it you'd just do:
    cabal configure --enable-target=bench --enable-target=test

    1. I think that it would be useful to have user defined targets (in some form) in *addition* to test suite and benchmark targets. The reason we want special targets that Cabal knows about is that Cabal can do special things to those targets.

      For example, Cabal can now (in theory) find, build, and run all test suites on Hackage and generate test coverage reports for all packages. Similarly for benchmarks. While it's not quite ready yet we're working on another type (as in the 'type' field) of test suite, where Cabal would have fine grained knowledge of every single test case and its result. This will lets us do even more fancy things with test suites.

  3. Nice! But how do I get it to produce a cool criterion report? I couldn't find a way to add '-o foo.html' to the command line. Hard-coding it seems an option, but I'm not sure I like that.

    1. You can use the --benchmark-option flag to cabal bench to pass options to the underlying binary (e.g. Criterion). --benchmark-option supports a number of template parameters e.g. the benchmark name etc that you can use. Example:

      cabal bench --benchmark-option=-o$benchmark

      Long term we want to add other 'type's to the benchmark section, which would allow Cabal to understand e.g. Criterion output natively and do interesting things with it.

    2. Thanks! For the time being, I decided to always produce a report. Those "long term" features sound great, especially if hackage (or a follow-on system) would show them for all packages.

  4. This comment has been removed by the author.

  5. Great post. I was hoping that cabal has support for benchmarking just as it has for testing and there it is :)

  6. One thought after using 'cabal bench' for a couple of days. It would be good if benchmark entry could reuse options from the executable/library section. Right now I have to duplicate all entries in ghc-options, cc-options and c-sources, which is both tedious and error prone.

    1. Your benchmark section can depend on your library section, which means that the library will be compiled using whatever options are specified in the library section (i.e. the library section is compiled first, then the benchmark section). Unless the benchmark section itself needs e.g. C sources, you can typically get away with just specifying -O2 or so.

    2. I didn't know that I can do that, but even if I make my benchmarks depend on the library the problem remains. I have lots of optimization options because of Repa and if I enable them only for the library and not for the benchmarking binary the performance degrades severely. So I still have to duplicate entries (not all though - now I can limit c-sources to library entry). It would be cool to be able to use a sort of "define" directive, but I guess this would have to be implemented as a general cabal feature, not limited to benchmarking.

    3. I could see how such a feature could be useful.

  7. Hi, is there a possibility to automatically save results in a file to make regression tests automatic, currently it seems that I should call cabal bench on old revision then on current. And it's not good.

    1. If you use Criterion to run your benchmarks you can use --benchmark-option='-u output.csv' to pass the "output results to file" flag to Criterion.

    2. You can specify these options by passing them in a Config object directly to your main function.