Prefect benchmarks

Benchmark conclusions

Conclusions for each benchmark run for all engines in all languages and settings

Summary

Prefect outperformed Airflow in most benchmarks, especially in long-running tasks, but still exhibited noticeable assignment delays and orchestration latency. It’s adequate for moderate workloads, but not optimized for high-frequency or highly parallel use cases.

Prefect setup

We set up Prefect version 2.14.4. We wrote our own simple docker compose since we couldn't find a recommended one in Prefect's documentation. We chose to use Postgresql as a database, as it is the recommended option for production usecases.

version: '3.8'

services:
  postgres:
    image: postgres:14
    restart: unless-stopped
    volumes:
      - db_data:/var/lib/postgresql/data
    expose:
      - 5432
    environment:
      POSTGRES_PASSWORD: changeme
      POSTGRES_DB: prefect
    healthcheck:
      test: ['CMD-SHELL', 'pg_isready -U postgres']
      interval: 10s
      timeout: 5s
      retries: 5

  prefect-server:
    image: prefecthq/prefect:2-latest
    command:
      - prefect
      - server
      - start
    ports:
      - 4200:4200
    depends_on:
      postgres:
        condition: service_started
    volumes:
      - ${PWD}/prefect:/root/.prefect
      - ${PWD}/flows:/flows
    environment:
      PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://postgres:changeme@postgres:5432/prefect
      PREFECT_LOGGING_SERVER_LEVEL: INFO
      PREFECT_API_URL: http://localhost:4200/api
volumes:
  db_data: null

The flow was defined using the following Python file.

from prefect import flow, task

ITER = 10     # respectively 40
FIBO_N = 33   # respectively 10

def fibo(n: int):
    if n <= 1:
        return n
    else:
        return fibo(n - 1) + fibo(n - 2)

@task
def fibo_task():
    return fibo(FIBO_N)

@flow(name="bench_{}".format(ITER))
def benchmark_flow():
    for i in range(ITER):
        fibo_task()

if __name__ == "__main__":
    benchmark_flow.serve(name="bench_{}".format(ITER))

Fibonacci 40 iterations, n=10

This benchmark measures the performance of Prefect when handling a large number of lightweight tasks. The tasks are Fibonacci calculations with 40 iterations and a base value of 10.

Python

Visualization

Statistics

	Prefect	Windmill	Windmill Dedicated
Total duration (in seconds)	4.872	4.383	2.092
Assignment	2.174 (44.62%)	1.836 (41.89%)	1.795 (85.80%)
Execution	1.546 (31.73%)	2.215 (50.54%)	0.122 (5.83%)
Transition	1.152 (23.65%)	0.332 (7.57%)	0.175 (8.37%)

Timing details

View task timing details

Task	Created at	Started at	Completed at
task_00	0.000	1.213	1.257
task_01	1.294	1.321	1.362
task_02	1.394	1.423	1.463
task_03	1.496	1.522	1.558
task_04	1.587	1.612	1.647
task_05	1.676	1.700	1.738
task_06	1.767	1.791	1.828
task_07	1.858	1.882	1.943
task_08	1.974	1.998	2.037
task_09	2.068	2.093	2.131
task_10	2.162	2.188	2.228
task_11	2.260	2.292	2.330
task_12	2.359	2.382	2.420
task_13	2.449	2.476	2.517
task_14	2.548	2.573	2.612
task_15	2.640	2.670	2.713
task_16	2.742	2.765	2.800
task_17	2.828	2.851	2.886
task_18	2.916	2.940	2.975
task_19	3.004	3.028	3.066
task_20	3.095	3.119	3.156
task_21	3.187	3.211	3.247
task_22	3.276	3.299	3.335
task_23	3.364	3.389	3.427
task_24	3.462	3.489	3.528
task_25	3.557	3.579	3.613
task_26	3.641	3.664	3.699
task_27	3.726	3.751	3.788
task_28	3.817	3.839	3.873
task_29	3.900	3.921	4.004
task_30	4.033	4.059	4.094
task_31	4.123	4.151	4.185
task_32	4.211	4.234	4.267
task_33	4.293	4.315	4.349
task_34	4.377	4.404	4.442
task_35	4.470	4.492	4.526
task_36	4.555	4.577	4.611
task_37	4.638	4.661	4.696
task_38	4.726	4.749	4.784
task_39	4.814	4.838	4.872

Windmill Comparison

Task	Created at	Started at	Completed at
task_00	0.000	0.003	0.059
task_01	0.067	0.113	0.171
task_02	0.180	0.226	0.280
task_03	0.290	0.335	0.389
task_04	0.398	0.446	0.501
task_05	0.510	0.558	0.614
task_06	0.622	0.669	0.725
task_07	0.732	0.780	0.834
task_08	0.842	0.889	0.942
task_09	0.950	0.997	1.052
task_10	1.061	1.108	1.166
task_11	1.175	1.220	1.274
task_12	1.283	1.330	1.385
task_13	1.394	1.440	1.494
task_14	1.503	1.550	1.605
task_15	1.612	1.661	1.716
task_16	1.723	1.770	1.823
task_17	1.831	1.878	1.930
task_18	1.939	1.986	2.041
task_19	2.049	2.096	2.152
task_20	2.161	2.209	2.266
task_21	2.274	2.320	2.376
task_22	2.384	2.431	2.486
task_23	2.495	2.542	2.596
task_24	2.604	2.652	2.706
task_25	2.715	2.761	2.816
task_26	2.825	2.872	2.925
task_27	2.933	2.979	3.033
task_28	3.042	3.090	3.145
task_29	3.154	3.201	3.269
task_30	3.278	3.325	3.382
task_31	3.391	3.437	3.493
task_32	3.501	3.548	3.602
task_33	3.611	3.660	3.715
task_34	3.723	3.770	3.823
task_35	3.833	3.879	3.934
task_36	3.942	3.990	4.045
task_37	4.053	4.101	4.157
task_38	4.165	4.212	4.268
task_39	4.277	4.324	4.383

Windmill Dedicated Comparison

Task	Created at	Started at	Completed at
task_00	0.000	0.019	0.022
task_01	0.029	0.073	0.077
task_02	0.081	0.125	0.128
task_03	0.134	0.179	0.182
task_04	0.187	0.231	0.234
task_05	0.239	0.284	0.287
task_06	0.292	0.338	0.341
task_07	0.345	0.391	0.394
task_08	0.398	0.444	0.447
task_09	0.451	0.497	0.500
task_10	0.505	0.549	0.552
task_11	0.557	0.603	0.606
task_12	0.610	0.655	0.659
task_13	0.663	0.709	0.712
task_14	0.716	0.761	0.764
task_15	0.768	0.814	0.817
task_16	0.821	0.867	0.870
task_17	0.876	0.921	0.924
task_18	0.929	0.973	0.976
task_19	0.981	1.027	1.030
task_20	1.035	1.080	1.083
task_21	1.087	1.132	1.135
task_22	1.139	1.186	1.189
task_23	1.193	1.238	1.241
task_24	1.246	1.292	1.295
task_25	1.299	1.345	1.348
task_26	1.352	1.398	1.401
task_27	1.405	1.451	1.454
task_28	1.458	1.504	1.507
task_29	1.512	1.557	1.560
task_30	1.564	1.611	1.614
task_31	1.618	1.664	1.667
task_32	1.671	1.717	1.720
task_33	1.724	1.770	1.773
task_34	1.777	1.823	1.826
task_35	1.830	1.876	1.879
task_36	1.884	1.930	1.933
task_37	1.937	1.983	1.986
task_38	1.991	2.036	2.039
task_39	2.043	2.089	2.092

Fibonacci 10 iterations, n=33

This benchmark measures the performance of Prefect when handling a small number of heavyweight tasks. The tasks are Fibonacci calculations with 10 iterations and a base value of 33.

Python

Visualization

Statistics

	Prefect	Windmill	Windmill Dedicated
Total duration (in seconds)	15.489	8.347	7.701
Assignment	1.513 (9.77%)	0.428 (5.13%)	0.370 (4.80%)
Execution	13.658 (88.18%)	7.832 (93.83%)	7.205 (93.56%)
Transition	0.318 (2.05%)	0.087 (1.04%)	0.126 (1.64%)

Timing details

View task timing details

Task	Created at	Started at	Completed at
task_00	0.000	1.270	2.629
task_01	2.673	2.703	4.059
task_02	4.095	4.121	5.475
task_03	5.508	5.534	6.916
task_04	6.951	6.979	8.337
task_05	8.373	8.401	9.816
task_06	9.849	9.874	11.253
task_07	11.287	11.313	12.675
task_08	12.710	12.737	14.070
task_09	14.102	14.129	15.489

Windmill Comparison

Task	Created at	Started at	Completed at
task_00	0.000	0.002	0.846
task_01	0.858	0.906	1.705
task_02	1.715	1.761	2.539
task_03	2.548	2.595	3.365
task_04	3.375	3.421	4.206
task_05	4.215	4.263	5.033
task_06	5.042	5.089	5.857
task_07	5.866	5.913	6.684
task_08	6.693	6.740	7.519
task_09	7.529	7.579	8.347

Windmill Dedicated Comparison

Task	Created at	Started at	Completed at
task_00	0.000	0.023	0.745
task_01	0.776	0.797	1.518
task_02	1.546	1.571	2.292
task_03	2.298	2.340	3.057
task_04	3.063	3.114	3.845
task_05	3.874	3.889	4.608
task_06	4.614	4.661	5.380
task_07	5.385	5.433	6.151
task_08	6.158	6.208	6.925
task_09	6.933	6.981	7.701