Semgrep, patterns instead of regexes and the Pro boundary

Semgrep, patterns instead of regexes and the Pro boundary

Semgrep matches rules against the syntax tree rather than against text, so renaming a variable or rewrapping a line does not break the pattern. Version 1.174.0, released on 20 August 2026, ships under LGPL 2.1, but only the engine is open. Cross-file analysis, three languages, secret detection and the rule registry require an account, and in most cases a payment as well.

How syntax matching works

Plain grep works on characters. If you are looking for a call with a dangerous argument, you have to anticipate every spelling: spaces around the comma, keyword arguments in a different order, a call split across three lines, a variable named cmd in one place and command in another. A regular expression covering all of that becomes unreadable after a few iterations and still misses the fourth variant.

Semgrep parses the file into a syntax tree and compares it with a pattern written in the syntax of the target language. The pattern subprocess.$FUNC(..., shell=$TRUE, ...) reads like Python because it is Python, with two additions. $FUNC and $TRUE are metavariables that capture any expression and remember it for later conditions. The three dots are the ellipsis operator, which matches any number of arguments, statements or elements in that position. Formatting, comments and the actual variable names stop mattering.

The shortest way to test the idea does not require a rule file.

# install into a dedicated virtual environment
python3 -m venv .venv && .venv/bin/pip install semgrep==1.174.0

# a one-off pattern with no rule file
semgrep --lang python --pattern 'subprocess.$FUNC(..., shell=True, ...)' src/

# a pattern with a replacement, preview mode
semgrep --lang python --pattern 'assert $X' --replacement 'if not $X: raise AssertionError' --dryrun src/

# local rules only, no network and no telemetry
semgrep scan --config ./rules --metrics off .

# list the files that will actually reach the scan
semgrep scan --config ./rules --x-ls .

The --metrics flag takes auto, on and off. The default auto means, according to the comment in metrics.py, that data is sent only when the configuration was pulled from the server. A scan using local rules sends nothing. A scan with --config p/default sends, and that is the default behaviour, easy to forget when introducing the tool at a company with a strict data policy.

Version, licence and the metadata that stays silent

The current release is 1.174.0. The source archive reached PyPI on 20 August 2026 at 15:59 UTC, the wheels a few minutes later, and the release entry in the repository carries a timestamp of 15:58 UTC the same day. Nine releases appeared between 11 June and 20 August 2026, an average of one every nine days, with a single longer gap of sixteen days across the turn of July and August. The registry holds 353 versions and not one is marked as yanked. The last change on the develop branch dates from 20 August 2026.

The licence is unambiguous only once you look in the right place. The LICENSE file at the repository root under tag v1.174.0 contains the full text of the GNU Lesser General Public License version 2.1 of February 1999, 504 lines. cli/LICENSE is a symbolic link to the same file, and its raw content on the hosting service is literally ../LICENSE. Beyond that the repository has no COPYING, no NOTICE, no LICENSING.md and no separate licence files in src, libs, languages, tools or interfaces.

The trouble starts in the registry metadata. pyproject.toml declares license = "LGPL-2.1-or-later" and license-files = ["LICENSE"], the new form from PEP 639. As a result PKG-INFO carries Metadata-Version: 2.4 and the field License-Expression: LGPL-2.1-or-later. The old License field is empty, and in the PyPI JSON interface info.license returns null. Worse, the classifier list contains not a single entry starting with License ::. There are eleven classifiers and they describe only the operating system, the Python versions and the topic category.

The practical consequence is that a dependency scanner reading info.license or the classifiers will see nothing and report the package as unlicensed. A scanner reading license_expression or PKG-INFO will see the correct LGPL-2.1-or-later. If your company runs an automated gate blocking packages without a licence, this particular one will trip it despite correctly labelled code. There is also a small inconsistency worth recording: the headers in the source files speak of licence version 2.1 without any later-version clause, while the metadata expression reads or-later. In a careful compliance audit that is a difference to settle with the legal team, not with the documentation.

The contents of the published package confirm the declaration, but they also reveal a split into two parts. The source archive weighs 507,637 bytes and contains Python code only: the semgrep and semdep packages, the LICENSE file with the LGPL text, pyproject.toml and README.md. The engine is not there. The engine is in the wheels. The wheel for macOS on arm64 weighs 49,480,382 bytes compressed and contains the file semgrep/bin/semgrep-core at 208,085,424 bytes, about 198 MiB, together with the libtree-sitter, libpcre2-8, libgmp, libzstd, libdwarf and libev libraries. Next to it sits semgrep-1.174.0.dist-info/licenses/LICENSE at 26,526 bytes, the same LGPL 2.1 text. The declaration matches the contents in full, and the code really is inside the package.

The runtime dependencies are numerous: twenty-seven packages plus pywin32 on Windows. Two carry exact pins, mcp==1.29.0 and ruamel.yaml.clib==0.2.15, and the required Python is at least 3.10. The pyproject.toml file itself recommends in a comment a separate virtual environment or a container image, and that is not overcaution, because exact pins on two libraries can block installation in an environment shared with another tool.

The line between the open and the paid version

This is where most writing about Semgrep describes capabilities without mentioning that they cost money. The split is visible directly in the code, in engine.py, where the EngineType enumeration has four values: OSS, PRO_LANG, PRO_INTRAFILE and PRO_INTERFILE.

The decide_engine_type method sets the default level by a simple rule: a logged-in user running semgrep ci gets PRO_INTRAFILE, everyone else gets OSS. A diff scan downgrades a requested PRO_INTERFILE to PRO_INTRAFILE so as not to slow down pull requests. A supply-chain-only scan does the same, because the cross-file mode defaults to a single worker.

The answer to the question in this section's title is therefore: cross-function and cross-file analysis is not in the open version. It lives in the semgrep-core-proprietary binary that the semgrep install-semgrep-pro command downloads. The code of that command is unambiguous. Without a token, meaning without semgrep login or without the SEMGREP_APP_TOKEN variable, the command exits with the invalid-key error code. The binary is fetched from the path api/agent/deployments/deepbinary/{platform} on the vendor's server. A 401 response means an invalid token, a 403 carries the message that the logged-in deployment has no access to the Pro Engine. A version stamp file lands next to the binary so that upgrading the package alone does not leave an incompatible engine behind.

Three further things are closed together with the engine. Secret detection raises an explicit exception in the code saying it is not part of the open source engine, and with an explicit --oss-only the scan aborts with a message that it is a proprietary extension. Dataflow tracing in the output, meaning --dataflow-traces, is available only for the PRO_INTRAFILE and PRO_INTERFILE levels. The languages marked as proprietary are exactly three: Apex, Elixir and Gosu. This comes from the lang.json file shipped with the package, where out of fifty entries only those three carry the is_proprietary tag.

The same file is worth reading for another reason. Language maturity is spread unevenly: eleven entries have status ga (C#, Go, JSON, Java, JavaScript, PHP, Python, Ruby, Scala, Terraform, TypeScript), one beta (Kotlin), twenty-five alpha and thirteen develop. The pricing page quotes "35+" supported languages for every plan. Both numbers are true within their own frame, since fifty entries also cover auxiliary items such as generic, regex and aliengrep plus separate variants for Python 2 and 3, but the gap is wide enough that when picking a tool for a specific stack it is better to check lang.json than the marketing page.

The rule registry is a service, not part of the open code. This is visible in config_resolver.py: identifiers with the r/, p/ and s/ prefixes (rule, pack, snippet) are turned into the address {semgrep_url}/c/{id}, and --config auto into {semgrep_url}/c/auto. The LGPL repository contains no production rules at all. The Community Edition rules live in a separate repository, semgrep-rules, and there I counted 2,091 identifiers across 2,076 YAML files, of which 378 for Python, 351 for Terraform, 257 for the generic mode, 182 for JavaScript, 132 in the directory covering AI libraries and 130 for Java.

The licence of that repository is a separate matter and the most commonly overlooked piece of the whole puzzle. The LICENSE file holds a single sentence pointing to the Semgrep Rules License v1.0, a document last updated on 13 December 2024. It is not an open licence. The grant is non-exclusive, royalty-free, worldwide, non-sublicensable and non-transferable. The limitation is phrased by purpose rather than by threshold: the rules may be used only for your own internal business purposes. The licence does not allow distributing them or making them available to others as a service. Licensing and copyright notices may not be removed or obscured, and copying a rule means carrying the notice with it. Modifications must be prominently marked in the copy. There is a patent clause that terminates immediately if you or your company makes a written claim that the rules or any Semgrep product infringe a patent. Breaching the terms ends the licence automatically.

What the licence does not contain matters just as much. There is no user-count threshold, no revenue threshold and no date of conversion into an open licence of the kind familiar from the Business Source License model. The restriction is permanent and concerns the kind of use. In practice: you may run these rules against your own code inside your company, you may not embed them in your own scanning product, you may not build a scanning service for clients on them and you may not publish a fork as an independent collection. Contributing to that repository grants Semgrep a licence to distribute the submitted rule on the same terms. Pro rules, written by the vendor's research team, never reach that repository at all and are available only through the registry.

To summarise the split: the open engine code is LGPL 2.1, the Pro engine is a closed binary behind a login, the Community Edition rules carry their own licence forbidding redistribution, and the Pro rules are proprietary and available only as a service.

Platform pricing and its arithmetic

The pricing page renders without JavaScript, so the figures below come from raw HTML fetched on 22 August 2026.

The Teams plan broken down by product: Code 30 USD, Supply Chain 30 USD, Secrets 15 USD, each per contributor per month. A team wanting all three therefore pays 75 USD per contributor per month, which at ten contributors comes to 750 USD per month and 9,000 USD per year. The free plan covers Code and Supply Chain at 0 USD, and Secrets does not appear in it.

Two things in this pricing deserve attention. The first is a discrepancy inside the page itself: the free plan description says "Scan up to 10 repositories", while the comparison table gives the same plan unlimited public repositories and at most ten private ones. I quote both, because the page does not let you settle which one applies.

The second is AI credits. The free plan gets sixty credits described as "included", with no billing period named. The Teams plan gets twenty per developer per month, so a three-person team reaches the same sixty credits, except every month. If the sixty on the free plan is a one-off pool, the gap between the plans is far wider than the raw numbers suggest. The page does not say.

A contributor is defined as someone who made at least one commit in the past ninety days to a private repository of the organisation scanned by Semgrep. With team turnover and with bots making commits, that definition can surprise you on the invoice.

One more fact looks like a contradiction and is not. The free platform plan includes cross-file analysis with Pro rules. That does not mean these features are in the open code. It means the vendor offers them free of charge to small teams through its platform, after logging in with GitHub or GitLab, capped at ten contributors. Free of charge and open are two different things here.

Writing your own rule

A custom rule is what separates Semgrep from an off-the-shelf scanner, and it is also the only reason to pick it over something simpler. The format is described by rule_schema_v1.yaml, shipped inside the package, so the field names below come from there rather than from the documentation.

A rule file has one top-level key, rules, holding a list. In search mode a rule requires id, message, languages, severity and one of the fields describing a pattern.

rules:
  - id: request-without-timeout
    message: >-
      A requests.$METHOD call without a timeout argument can hang the process
      indefinitely. Set timeout explicitly.
    languages: [python]
    severity: WARNING
    patterns:
      - pattern-either:
          - pattern: requests.get(...)
          - pattern: requests.post(...)
          - pattern: requests.request(...)
      - pattern-not: requests.$METHOD(..., timeout=$T, ...)
    paths:
      exclude:
        - tests/
        - "**/conftest.py"
    metadata:
      category: correctness
      confidence: HIGH

The order of operators inside patterns is a conjunction: every condition must hold at once. pattern-either is a disjunction. pattern-not subtracts the cases that are already correct. paths with its include and exclude subfields restricts the rule to part of the repository without touching the global configuration.

The severity field takes ERROR, WARNING and INFO, and since version 1.72.0 also CRITICAL, HIGH, MEDIUM and LOW. The schema additionally permits the experimental INVENTORY and EXPERIMENT. Old and new names coexist, so it is easy to end up with a mixture in one repository and a --severity filter catching half of what it should.

The second rule shows conditions applied to metavariables together with an automatic fix.

rules:
  - id: weak-jwt-signing-algorithm
    message: JWT signed with algorithm $ALG. Use HS256 or RS256.
    languages: [python]
    severity: ERROR
    min-version: 1.72.0
    patterns:
      - pattern: jwt.encode(..., algorithm=$ALG, ...)
      - metavariable-regex:
          metavariable: $ALG
          regex: ^["'](none|HS1|MD5)["']$
      - focus-metavariable: $ALG
    fix: '"HS256"'
    metadata:
      cwe:
        - "CWE-327: Use of a Broken or Risky Cryptographic Algorithm"

metavariable-regex applies a regular expression to the content captured by a metavariable, mixing both worlds in a controlled way: the syntax picks the place, the regex narrows the value. focus-metavariable moves the highlight in the result from the whole call to the argument alone, which lets the fix replace only that argument. Alongside these the schema defines metavariable-pattern (matching a pattern against a metavariable's content, possibly in another language), metavariable-comparison with the fields metavariable, comparison, strip and base, plus metavariable-type, metavariable-name and metavariable-analysis with the fields analyzer and metavariable, used among other things for entropy analysis. The min-version field exists so that an older Semgrep skips the rule instead of failing on an unknown field. The schema deliberately sets additionalProperties: true at rule level for the same reason.

The third variant is taint mode, the only way to connect a data source with the place the data is used.

rules:
  - id: os-command-from-flask-parameter
    mode: taint
    message: Request data reaches os.system without validation.
    languages: [python]
    severity: ERROR
    pattern-sources:
      - patterns:
          - pattern-inside: |
              @app.route(...)
              def $HANDLER(...):
                  ...
          - pattern: flask.request.$ANYTHING
    pattern-sanitizers:
      - pattern: shlex.quote(...)
      - pattern: re.fullmatch("...", ...)
    pattern-sinks:
      - pattern: os.system(...)

A rule in taint mode requires id, message, languages, severity, pattern-sources and pattern-sinks. pattern-sanitizers and pattern-propagators are optional. And here the split from the previous section returns: in the open engine, tracking ends at the function boundary. If the source is in a controller and the system call sits in a helper function in the same file, you need --pro-intrafile. If the helper lives in another module, you need --pro. A rule written without that awareness looks broken while being perfectly correct.

Rules are checked with semgrep --test, which looks for a file with the same base name and the language extension and compares the hits with comment annotations. The recognised ones are ruleid, ok, todoruleid and todook, plus the proruleid, deepruleid and deepok variants for findings that need the Pro engine. semgrep --validate --config ./rules checks the schema alone without running a scan.

False positives and the maintenance cost

Static analysis produces noise, and a team receiving a hundred warnings a day stops reading them within a week. Rolling out Semgrep is not a one-off configuration and it is better to assume that up front than to discover it in month three.

There are several suppression mechanisms and they differ in reach. The narrowest is a comment in the code. From constants.py it follows that the recognised token is nosem or nosemgrep, that case is ignored, and that after a colon you can list rule identifiers separated by commas. The comment works on the same line as the finding or on the line directly above it.

# nosemgrep: request-without-timeout
resp = requests.get(url)

resp = requests.get(url)  # nosem: request-without-timeout,weak-jwt-signing-algorithm

# nosemgrep
resp = requests.get(url)

The last variant, without an identifier, silences everything at that spot and is the one that tells nobody anything a year later. It pays to require a rule identifier and a reason during code review. The whole comment mechanism is disabled by --disable-nosem, which is useful in a periodic audit of accumulated suppressions.

One level up sits the .semgrepignore file with path-pattern syntax, complemented by default with .gitignore unless you pass --no-git-ignore. There is a catch visible in target_manager.py: for some products the .semgrepignore file is deliberately ignored, because its typical content filters out directories that those very products must scan. The file name is changed by --x-semgrepignore-filename and skipping it by --x-ignore-semgrepignore-files. The --x- prefix marks an experimental flag, so I would not build a permanent process on it.

The broadest level is rule and threshold selection. --exclude-rule drops a rule by identifier, --exclude and --include filter paths, and --severity lets through a chosen level only. The most effective in practice, though, is --baseline-commit: the scan reports only findings that were absent at the given point in history. An existing repository therefore does not start with a thousand warnings, while new code is guarded from day one.

On top of that come performance limits that are easy to trip over. The default time budget per rule per file is five seconds, and a file larger than one million bytes is skipped. Cross-file mode in a continuous integration pipeline has its own defaults: a memory limit of 8,192 MiB and a time limit of 10,800 seconds, that is three hours. On Linux the memory limit is additionally computed as ninety percent of the value in /sys/fs/cgroup/memory.max when such a limit exists. A scan that "found nothing" is sometimes a scan that quietly skipped half the files, which is why --x-ls and the skipped-files section of the report are the first place to check.

An honest cost forecast looks like this, with the caveat that the times below are my own estimate from practice rather than a figure from the documentation. The first run on a mature repository takes a good quarter of an hour. Reviewing the output and setting a baseline takes a few days. After that comes a standing overhead: every registry update may add rules that produce noise in your code, every new custom rule needs a test and a review, and every suppression needs a justification, otherwise a year later nobody knows whether a nosemgrep comment marks a real exception or a moment of impatience.

Semgrep next to linters and model scanners

The line between a linter and a security tool blurs, since both read code without running it. The difference lies in the goal and in what they are able to see.

Biome and Oxlint guard style and common slips in JavaScript and in TypeScript. They run faster because their scope is narrower and they work on a single file without a dataflow model. Semgrep will not replace them for formatting and will not check types, while they will not express a rule describing data flowing from a request parameter into a system call. A sensible arrangement is linting on file save and Semgrep in the continuous integration pipeline, with the baseline set to the main branch.

Tools such as Lakera Guard, Prompt Security and Protect AI belong to a different category, and mixing them with Semgrep leads to poor decisions. They inspect what goes into a language model and what comes out of it, at runtime. Semgrep inspects code before it runs at all. The single point of contact is that the Community Edition rules repository contains an ai directory with 132 rules covering libraries for working with models, and that is analysis of the code calling a model, not of the traffic to the model.

A separate matter is code written by assistants. If your team uses GitHub Copilot, a pattern scan in the pipeline is worth more than it used to be, because the volume of code passing through review grows faster than reviewer attention. Semgrep 1.174.0 also ships its own MCP server in the semgrep.mcp module and pulls in the mcp==1.29.0 dependency, so integration with agent tooling is in the package, though I did not verify its description beyond the presence of the code.

Common mistakes

The first is assuming --config auto works locally and offline. That shorthand expands to the address {semgrep_url}/c/auto, requires a connection to the vendor's server and enables telemetry, since the default --metrics auto fires on any remotely fetched configuration.

The second is writing a rule in taint mode and concluding the engine is broken because it finds nothing. Without --pro-intrafile the tracking stops at the function boundary, and that covers a minority of real cases.

The third is copying rules from the semgrep-rules repository into your own product or your own public collection. The licence on those rules forbids redistribution and offering them as a service, regardless of the repository being public on GitHub.

The fourth is a block in a compliance gate over an alleged missing licence. The package carries a correct LGPL-2.1-or-later declaration, only in the License-Expression field rather than in the classifiers or the old License field. The fix is updating the scanner, not adding an exception for the package.

The fifth is installing it alongside other tools in one virtual environment. The exact pins mcp==1.29.0 and ruamel.yaml.clib==0.2.15 clash with other packages more often than you would expect.

The sixth is running a scan across the whole repository with no baseline. The output on a mature project runs into hundreds of findings, the team closes it with a single blanket suppression, and the tool is dead. --baseline-commit exists for exactly this.

The seventh is mixing old and new severity names in one rule set. ERROR and HIGH are two distinct values in the schema, and the --severity filter knows of no relationship between them.

FAQ

Is Semgrep free?

The engine and the command line client are LGPL 2.1 and usable without an account. Cross-function analysis, cross-file analysis, secret detection and three languages (Apex, Elixir, Gosu) require the Pro binary downloaded after logging in. The platform has a free plan for at most ten contributors and paid plans from 30 USD per contributor per month.

Can I use rules from the semgrep-rules repository in my own tool?

No. Semgrep Rules License v1.0 permits use of the rules only for your own internal business purposes and explicitly forbids distributing them or making them available to others as a service. There is no revenue threshold and no date of conversion to an open licence.

How does Semgrep differ from a linter?

A linter guards style and common slips using a built-in rule set. Semgrep matches any pattern you write yourself in the syntax of the target language, and it can trace data flow from a source to the place it is used. In exchange it does not format code and does not check types.

Does a scan send my code to Semgrep?

Code does not leave the machine on a local scan. Scan metadata is sent, and only when the rules came from the server. --metrics off disables that entirely, and --config ./rules with local files never triggers a send at all.

How many rules do I get without an account?

The registry works anonymously for public packs with the p/, r/ and s/ prefixes. The GitHub counterpart of that set is the Community Edition repository, where I counted 2,091 rules. Pro rules are not in that repository and require an account.

Is it worth writing custom rules, or are the stock ones enough?

Stock rules catch common vulnerabilities and require no work. Custom ones make sense where the point is a specific project's conventions: banning a deprecated internal function, requiring a request context to be passed through, catching a pattern ahead of a refactor. That is the one part an off-the-shelf scanner cannot replace.