Benchmarking Opengrep Performance Improvements
Opengrep's improvements to rule load times resulted in 3.15x faster average scan times than Semgrep
Opengrep's improvements to rule load times resulted in 3.15x faster average scan times than Semgrep
Opengrep's improvements to rule load times resulted in 3.15x faster average scan times than Semgrep
Opengrep's improvements to rule load times resulted in 3.15x faster average scan times than Semgrep
Opengrep's improvements to rule load times resulted in 3.15x faster average scan times than Semgrep
Opengrep is an open-source tool originating from Semgrep and designed to scan code and detect vulnerable code patterns based on predefined rules. It launched earlier this year with support from vendors across the application security industry, including Endor Labs.
Since the initial release of Opengrep 1.0.0 in March 2025, many new features have been introduced, including Windows support and more comprehensive Sarif output. The team has also invested work in performance improvements for the project, starting with optimizing rule load times.
Performance is critical for a tool that may need to scan thousands of company repositories and run on each pull request (PR). Following Opengrep’s announcement of performance improvements—particularly in rule loading—we conducted tests to evaluate the impact on overall scan times .
We used Semgrep 1.122.0 as a comparison benchmark to understand how the two projects were diverging over time. It is important to note that our experiments focused on running Semgrep and Opengrep with rule sets loaded locally. This reflects a common usage pattern in both the community and industry, particularly in scenarios where custom rules are developed and applied.
Our results show that Opengrep, on average, was 3.15x faster than Semgrep, with performance gains ranging from 1.07x to 4.14x depending on the size of the project. Performance improvements were less significant for projects with more than 200,000 lines of code where scan time has a greater role than rule load time.
Methodology
The scanning process includes the following steps:
- Loading rules: Load SAST rule configurations from specified sources.
- Code processing: Identify target files and parse them into intermediate representations, such as abstract syntax trees (ASTs), for analysis.
- Analysis: Match patterns against ASTs and evaluate metavariables to identify potential vulnerabilities and issues.
- Results processing: Filter findings based on rule configurations and format the output in the requested format (JSON, SARIF, text).
Two key factors significantly affect performance:
- Number of rules and complexity: The more rules, the more the time to load and run them. The pattern complexity of these rules (metavariables, taint-mode, etc.) also influences performance.
- Codebase size and programming language: Larger codebases with more files and lines of code naturally take longer to process. The programming language also influences the performance, as some languages are more complex to parse than others.
In order to perform a fair comparison between Opengrep and Semgrep, we selected a common ruleset, and we ran on the same target applications ranging over different sizes and programming languages. To reduce noise, each project was scanned 10 times and we calculated the average scan time.
- Machine: We used a high-performance Azure cloud virtual machine running Linux kernel 6.11 on x86_64 architecture with 8 physical CPU cores (16 logical cores with hyperthreading) and 62.75 GB of RAM.
- Rules: We utilized the official rule collection from the Semgrep repository, excluding directories that failed Semgrep's validation command (i.e., semgrep –-validate). This minimizes bias while maintaining testing integrity. Although Semgrep should theoretically perform optimally on its native ruleset, this approach provides a fair evaluation framework since Opengrep was derived from the same codebase. Using Semgrep's own rules creates a controlled environment where any performance differences can be more meaningfully attributed to the underlying scanning technology rather than rule optimization. Overall, 1820 rules were selected spanning over more than 15 programming languages.
- Target applications: We selected applications spanning seven upstream programming languages with varying sizes and complexity, from large projects with thousands of files and hundreds of thousands lines of code (LoC) to small projects with less than a hundred files and a few hundred LoC.
Results
Opengrep 1.2.2 averaged a 3.15x speed improvement over Semgrep 1.122.0, with a median gain of 3.22x across all projects. Both tools delivered consistent scan times across multiple runs, with minimal variation in performance between runs.
While both tools generally demonstrate consistency in reporting findings, we observed a few exceptions along two dimensions:
- Cross-tool variation: Despite using the same ruleset, Opengrep and Semgrep did not always produce identical results. This discrepancy occurred in 8 out of 19 target applications. In most cases, the difference was minor—less than 1% of the total findings—and typically attributable to a small subset of rules (fewer than 10).
- Intra-tool variation: Both tools occasionally reported slightly different numbers of findings across repeated runs on the same codebase. These variations were also small, with a coefficient of variation below 1% in most cases, and again linked to a limited number of rules (fewer than 10).
To the best of our knowledge, such inconsistencies are expected and can be attributed to several known factors:
- Heuristic optimizations used by the tools to speed up analysis,
- Non-deterministic behavior in certain complex rules (e.g., taint analysis or dataflow, which may rely on timeouts or approximation), and
- Execution order effects or differences in parallelism during rule evaluation.
Spot-checking the findings where the tools disagreed revealed that in some cases Opengrep was correct, while in others Semgrep produced the accurate result.
Conclusion
Opengrep showed notable improvement in rule load times, resulting in significant performance improvements for projects with small to medium codebases. Those gains were less significant for projects with larger codebases, and performance improvements decreased as the codebase grew. For teams scanning small to medium codebases or integrating security into CI/CD workflows, Opengrep’s rule loading gains translate into meaningful improvements with shorter pipeline runtimes and reduced infrastructure costs.
Since its launch in March 2025, Opengrep has seen rapid progress with 26 releases and 61 contributors from the community to date. Opengrep is emerging as a compelling choice for efficient, developer-friendly application security testing as it continues to invest in performance improvements, platform support, and reporting.
