futhark-bench

SYNOPSIS

futhark-bench [–runs=count | –compiler=program | –json | –no-validate] programs…

DESCRIPTION

This tool is the recommended way to benchmark Futhark programs. Programs are compiled using the specified compiler (futhark-c by default), then run a number of times for each test case, and the average runtime printed on standard output. A program will be ignored if it contains no data sets - it will not even be compiled. Only test cases that use the default entry point (main) are considered.

If compilation or running fails, an error message will be printed and benchmarking will continue (and --json will write the file), but a non-zero exit code will be returned at the end.

OPTIONS

--runs=count The number of runs per data set.
--compiler=program
 The program used to compile Futhark programs. This option can be passed multiple times, resulting in multiple compilers being used for each test case. The specified program must support the same interface as futhark-c.
--json=file Write raw results in JSON format to the specified file.
--pass-option=opt
 

Pass an option to benchmark programs that are being run. For example, we might want to run OpenCL programs on a specific device:

futhark-bench prog.fut --compiler=futhark-opencl --pass-option=-dHawaii
--timeout=seconds
 

If the runtime for a dataset exceeds this integral number of seconds, it is aborted. Note that the time is allotted not per run, but for all runs for a dataset. A twenty second limit for ten runs thus means each run has only two seconds (minus initialisation overhead).

A negative timeout means to wait indefinitely.

--skip-compilation
 Do not run the compiler, and instead assume that each benchmark program has already been compiled. Use with caution.
--exclude-case=TAG
 Do not run test cases that contain the given tag. Cases marked with “nobench” or “disable” are ignored by default.

EXAMPLES

The following program benchmarks how quickly we can sum arrays of different sizes:

-- How quickly can we reduce arrays?
--
-- ==
-- nobench input { 0 }
-- output { 0 }
-- input { 100 }
-- output { 4950 }
-- compiled input { 100000 }
-- output { 704982704 }
-- compiled input { 100000000 }
-- output { 887459712 }

let main(n: i32): i32 =
  reduce (+) 0 (iota n)

SEE ALSO

futhark-c(1), futhark-test(1)