220112.1 · 1c14cd25038647f939cae9c1c07ba2589aa97d61
released 3.6.0 new option --replace & faster searching
reading declarations
@@ -1,6 +1,6 @@ [![build status][travis-image]][travis-url] [![Language grade: C/C++][lgtm-image]][lgtm-url] [![license][bsd-3-image]][bsd-3-url] -**ugrep v3.5 is now available: more features & even faster than before**+**ugrep v3.6 is now available: more features & even faster than before** Search for anything in everything... ultra fast @@ -10,6 +10,8 @@ Search for anything in everything... ultra fast - Supports all GNU/BSD grep standard options; a compatible replacement for GNU/BSD grep, [see how](#grep) +- Built-in help facility: `ugrep --help WHAT` displays options related to `WHAT` you specified+ - Matches Unicode patterns by default in UTF-8, UTF-16, UTF-32 encoded files - Ultra fast with new match algorithms and features beating grep, ripgrep, silver searcher, ack, sift, etc.@@ -118,10 +120,19 @@ Search for anything in everything... ultra fast ugrep --csv PATTERN ... ugrep --json PATTERN ... ugrep --xml PATTERN ... ugrep --format='file=%f line=%n match=%O%~' PATTERN ... + > **Pro Tip:** `ugrep --help format` displays help on format `%` fields.+ - Search with PCRE's Perl-compatible regex patterns and display or replace [subpattern matches](#replace) ugrep -P PATTERN ... ugrep -P --format='%1 and %2%~' 'PATTERN(SUB1)(SUB2)' ... +- Replace patterns in the output with [-P and --replace](#replace) text with `%` [formatting fields](#format), optionally use `-y` to pass the rest of the file through:++ ugrep --replace='(%m:%o)' PATTERN ... ugrep -y --replace='(%m:%o)' PATTERN ...+ ugrep -P --replace='%1' PATTERN ... ugrep -y -P --replace='%1' PATTERN ...++ > **Pro Tip:** `ugrep --help format` displays help on format `%` fields.+ - Search files with a specific [encoding](#encoding) format such as ISO-8859-1 thru 16, CP 437, CP 850, MACROMAN, KOI8, etc. ugrep --encoding=LATIN1 PATTERN ...@@ -168,7 +179,7 @@ Table of contents - [Displaying colors with --color and paging the output with --pager](#color) - [Output matches in JSON, XML, CSV, C++](#json) - [Customize output with --format](#format)- - [Replacing matches with -P --format backreferences to group captures](#replace)+ - [Replacing matches with -P --replace and --format using backreferences](#replace) - [Limiting the number of matches with -1,-2...-9, -K, -m, and --max-files](#max) - [Matching empty patterns with -Y](#empty) - [Case-insensitive matching with -i and -j](#case)@@ -325,8 +336,8 @@ To build `ugrep` with specific hard defaults enabled, such as a pager: Options to select defaults for builds include: -- `--enable-hidden` search hidden files and directories-- `--enable-pager` use a pager to display output on terminals+- `--enable-hidden` always search hidden files and directories+- `--enable-pager` always use a pager to display output on terminals - `--enable-pretty` colorize output to terminals and add filename headings - `--disable-auto-color` disable automatic colors, requires ugrep option `--color=auto` to show colors - `--disable-mmap` disable memory mapped files@@ -450,17 +461,17 @@ input file is too large (T3: 13GB file) resulting in an error. GREP | T1 | T2 | T3 | T4 | T5 | T6 | T7 | T8 | T9 | T10 | T11 | T12 | --------------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |-ugrep | **0.03** | **0.04** | **5.06** | **0.07** | **0.02** | **0.98** | **0.97** | **0.87** | **0.26** | **0.10** | **0.19** | **0.02** |+ugrep | **0.03** | **0.04** | **6.05** | **0.07** | **0.02** | **0.98** | **0.97** | **0.87** | **0.26** | **0.10** | **0.19** | **0.02** | hyperscan grep | 0.09 | 0.10 | **4.35** | 0.11 | 0.04 | 7.78 | 3.39 | 1.41 | 1.17 | *n/a* | *n/a* | *n/a* | ripgrep | 0.06 | 0.10 | 7.50 | 0.19 | 0.06 | 2.20 | 2.07 | 2.01 | 2.14 | 0.12 | 0.36 | 0.03 | silver searcher | 0.10 | 0.11 | *n/a* | 0.16 | 0.21 | *n/a* | *n/a* | *n/a* | *n/a* | 0.45 | 0.32 | 0.09 |-GNU grep 3.3 | 0.08 | 0.15 | 11.26 | 0.18 | 0.16 | 2.70 | 2.64 | 2.42 | 2.26 | *n/a* | 0.26 | *n/a* |+GNU grep 3.3 | 0.08 | 0.15 | 11.21 | 0.18 | 0.16 | 2.70 | 2.64 | 2.42 | 2.26 | *n/a* | 0.26 | *n/a* | PCREGREP 8.42 | 0.17 | 0.17 | *n/a* | 0.26 | 0.08 | *n/a* | *n/a* | *n/a* | *n/a* | *n/a* | 2.37 | *n/a* | BSD grep 2.5.1 | 0.81 | 1.60 | *n/a* | 1.85 | 0.83 | *n/a* | *n/a* | *n/a* | *n/a* | *n/a* | 3.35 | 0.60 | Note T3: [Hyperscan simplegrep](https://github.com/intel/hyperscan/tree/master/examples)-was compiled with optimizations enabled. Hyperscan results for T3 are slightly-better than ugrep, as expected because hyperscan simplegrep has one advantage+was compiled with optimizations enabled. Hyperscan results for T3 are somewhat+better than ugrep as expected because hyperscan simplegrep has one advantage here: it does not maintain line numbers and other line-related information. By contrast, line information should be tracked (as in ugrep) to determine if matches are on the same line or not, as required by option `-c`. Hyperscan@@ -701,20 +712,20 @@ and [`soffice`](https://www.libreoffice.org) to be installed. See ### Notable improvements over grep - **ugrep** starts an interactive query UI with option `-Q`.-- **ugrep** matches patterns across multiple lines.+- **ugrep** matches patterns across multiple lines when patterns match `\n`. - **ugrep** matches full Unicode by default (disabled with option `-U`). - **ugrep** supports fuzzy (approximate) matching with option `-Z`. - **ugrep** supports gitignore with option `--ignore-files`. - **ugrep** supports user-defined global and local configuration files. - **ugrep** supports Boolean patterns with AND, OR and NOT (option `--bool`).-- **ugrep** searches compressed files with option `-z`.+- **ugrep** searches compressed files and archives with option `-z`. - **ugrep** searches cpio, jar, pax, tar and zip archives with option `-z`. - **ugrep** searches cpio, jar, pax, tar and zip archives recursively stored- within archives with `-z` and `--zmax=NUM` for `NUM` recursive levels deep.+ within archives with `-z` and `--zmax=NUM` for up to `NUM` levels deep. - **ugrep** searches pdf, doc, docx, xls, xlsx, epub, and more with `--filter` using third-party format conversion utilities as plugins. - **ugrep** searches a directory when the FILE argument is a directory, like- most Unix/Linux utilities; option `-r` searches directories recursively.+ most Unix/Linux utilities; use option `-r` to search directories recursively. - **ugrep** does not match hidden files by default like most Unix/Linux utilities (hidden dotfile file matching is enabled with `-.`). - **ugrep** regular expression patterns are more expressive than GNU grep and@@ -976,6 +987,26 @@ considered both files and directories to add to `--include` and directory paths are not explicitly listed in this file then it will not be visited using `--include-from`. +Because ugrep checks if the input is valid UTF-encoded Unicode (unless `-U` is+used), it is possible to use it as a filter to ignore non-UTF output produced+by a program:++ program | ugrep -I ''++If the program produces valid output then the output is passed through,+otherwise the output is filtered out option `-I`. If the output is initially+valid for a very large portion but is followed by invalid output, then ugrep+may initially show the output up to but excluding the invalid output after+which further output is blocked.++To filter lines that are valid ASCII or UTF-encoded, while removing lines that+are not:++ program | ugrep '[\p{Unicode}--[\n]]+'++Note that `\p{Unicode}` matches `\n` but we don't want to matche the whole+file! Just lines with `[\p{Unicode}--[\n]]+`.+ 🔝 [Back to table of contents](#toc) <a name="help"/>@@ -1115,7 +1146,7 @@ This option starts a user interface to enter search patterns interactively: screen. If Alt/Meta keys are not supported (e.g. X11 xterm), then press CTRL-O followed by the key corresponding to the option. - Press Alt-g to enter or edit option `-g` file and directory matching globs, a- comma-separated list of gitignore-style glob patterns. Press ESC to return+ comma-separated list of gitignore-style glob patterns. Presssing ESC returns control to the query pattern prompt (the globs are saved). When a glob is preceded by a `!` or a `^`, skips files whose name matches the glob When a glob contains a `/`, full pathnames are matched. Otherwise basenames are@@ -1123,7 +1154,7 @@ This option starts a user interface to enter search patterns interactively: - The query UI prompt switches between `Q>` (normal), `F>` (fixed strings), `G>` (basic regex), `P>` (Perl matching), and `Z>` (fuzzy matching). When the `--glob=` prompt is shown, a comma-separated list of gitignore-style- glob patterns may be entered. Press ESC returns control to the pattern+ glob patterns may be entered. Presssing ESC returns control to the pattern prompt. - Press Enter to switch to selection mode to select lines to output when ugrep exits. Normally, ugrep in query mode does not output any results unless@@ -1137,11 +1168,11 @@ This option starts a user interface to enter search patterns interactively: - Press TAB to chdir one level down into the directory of the file listed or viewed at the top of the screen. If no directory exists, the file itself is selected to search. Press Shift-TAB to go back up one level.-- Press CTRL-T to toggle colors on or off. Normally ugrep in query mode uses- colors and other markup to highlight the results. When colors are turned- off, selected results are also not colored in the output produced by ugrep- when ugrep exits. When colors are turned on (the default), selected results- are colored depending on the `--color` option.+- Press CTRL-T to toggle colors on and off. Normally ugrep in query mode uses+ colors and other markup to highlight results. When colors are turned off,+ selected results are also not colored in the output produced by ugrep when+ ugrep exits. When colors are turned on (the default), selected results are+ colored depending on the `--color` option. - The query engine is optimized to limit system load by performing on-demand searches to produce results only for the visible parts shown in the interface. That is, results are shown on demand, when scrolling down and@@ -1150,7 +1181,7 @@ This option starts a user interface to enter search patterns interactively: effectively limits the load on the system to maintain a high degree of responsiveness of the query engine to user input. Because the search results are produced on demand, occasionally you may notice a flashing "Searching..."- message when searching files on slower systems.+ message when searching files. - To display results faster, specify a low `DELAY` value such as 1. However, lower values may increase system load as a result of repeatedly initiating and cancelling searches by each key pressed.@@ -1937,14 +1968,14 @@ comments: zstd (requires suffix .zst, .zstd, .tzst). --zmax=NUM When used with option -z (--decompress), searches the contents of- compressed files and archives stored within archives for up to NUM- recursion levels deep. The default --zmax=1 only permits searching+ compressed files and archives stored within archives by up to NUM+ recursive expansions. The default --zmax=1 only permits searching uncompressed files stored in cpio, pax, tar and zip archives; compressed files and archives are detected as binary files and are effectively ignored. Specify --zmax=2 to search compressed files and archives stored in cpio, pax, tar and zip archives. NUM may range from 1 to 99 for up to 99 decompression and de-archiving- steps. Larger NUM values gradually degrade performance.+ steps. Increasing NUM values gradually degrades performance. Files compressed with gzip (`.gz`), compress (`.Z`), bzip2 (`.bz`, `.bz2`, `.bzip2`), lzma (`.lzma`), xz (`.xz`), lz4 (`.lz4`) and zstd (`.zst`, `.zstd`)@@ -2697,14 +2728,14 @@ glob | matches `/**` | when at the end of a glob, matches everything after the `/` `\?` | matches a `?` (or any character specified after the backslash) -When a glob contains a path separator `/`, the pathname is matched. Otherwise-the basename of a file or directory is matched. For example, `*.h` matches-`foo.h` and `bar/foo.h`. `bar/*.h` matches `bar/foo.h` but not `foo.h` and not-`bar/bar/foo.h`.+When a glob contains a path separator `/`, the full pathname is matched.+Otherwise the basename of a file or directory is matched in recursive searches.+For example, `*.h` matches `foo.h` and `bar/foo.h`. `bar/*.h` matches+`bar/foo.h` but not `foo.h` and not `bar/bar/foo.h`. When a glob begins with a `/`, files and directories are matched at the working-directory, i.e. not recursively. For example, use a leading `/` to force-`/*.h` to match `foo.h` but not `bar/foo.h`.+directory, not recursively. For example, use a leading `/` to force `/*.h` to+match `foo.h` but not `bar/foo.h`. When a glob ends with a `/`, directories are matched instead of files, same as `--include-dir`.@@ -3114,10 +3145,21 @@ To extract a table from an HTML file and put it in C/C++ source code using ### Customized output with --format --format=FORMAT- Output FORMAT-formatted matches. For example `--format=%f:%n:%O%~'+ Output FORMAT-formatted matches. For example --format='%f:%n:%O%~' outputs matching lines `%O' with filename `%f` and line number `%n'- followed by a newline `%~'. Context options -A, -B, -C, and -y are- ignored. See `man ugrep' section FORMAT.+ followed by a newline `%~'. When option -P is used, FORMAT may+ include `%1' to `%9', `%[NUM]#' and `%[NAME]#' to output group+ captures. A `%%' outputs `%'. See `ugrep --help format' and `man+ ugrep' section FORMAT for details. Context options -A, -B, -C and+ -y are ignored.+ -P, --perl-regexp+ Interpret PATTERN as a Perl regular expression.++Use option `-P` to use group captures and backreferences. Capturing groups in+regex patterns are parenthesized expressions `(pattern)` and the first is+referenced in `FORMAT` by `%1`, the second by `%2` and so on. Named captures+are of the form `(?<NAME>pattern)` and are referenced in `FORMAT` by+`%[NAME]#`. The following output formatting options may be used. The `FORMAT` string `%`-fields are listed in a table further below:@@ -3191,6 +3233,7 @@ field | output `%:` | if not the first match: a colon, same as `%[:]>` `%;` | if not the first match: a semicolon, same as `%[;]>` `%│` | if not the first match: a vertical bar, same as `%[│]>`+`%S` | if not the first match: separator, see also `%[SEP]$` `%[ARG]S` | if not the first match: `ARG` and separator, see also `%[SEP]$` `%s` | the separator, see also `%[ARG]S` and `%[SEP]$` `%~` | a newline character@@ -3212,7 +3255,7 @@ field | output `%e` | the ending byte offset of the match `%Z` | the edit distance cost of an approximate match with option `-Z` `%u` | select unique lines only unless option -u is used-`%1`,`%2`,...,`%9` | the first regex group capture of the match, and so on up to group `%9`, requires option `-P`+`%1` `%2` ... `%9` | the first regex group capture of the match, and so on up to group `%9`, requires option `-P` `%[NUM]#` | the regex group capture `NUM`; requires option `-P` `%[NUM]b` | the byte offset of the group capture `NUM`; requires option `-P` `%[NUM]e` | the ending byte offset of the group capture `NUM`; requires option `-P`@@ -3338,24 +3381,69 @@ the output: <a name="replace"/> -### Replacing matches with -P --format backreferences to group captures+### Replacing matches with -P --replace and --format using backreferences - --format=FORMAT- Output FORMAT-formatted matches. For example `--format=%f:%n:%O%~'- outputs matching lines `%O' with filename `%f` and line number `%n'- followed by a newline `%~'. See `man ugrep' section FORMAT.+ --replace=FORMAT+ Replace matching patterns in the output by the specified FORMAT+ with `%' fields. When option -P is used, FORMAT may include `%1'+ to `%9', `%[NUM]#' and `%[NAME]#' to output group captures. A `%%'+ outputs `%' and `%~' outputs a newline. See option --format,+ `ugrep --help format' and `man ugrep' section FORMAT for details+ -y, --any-line+ Any line is output (passthru). Non-matching lines are output as+ context with a `-' separator. See also options -A, -B, and -C. -P, --perl-regexp Interpret PATTERN as a Perl regular expression.+ --format=FORMAT+ Output FORMAT-formatted matches. For example --format='%f:%n:%O%~'+ outputs matching lines `%O' with filename `%f` and line number `%n'+ followed by a newline `%~'. When option -P is used, FORMAT may+ include `%1' to `%9', `%[NUM]#' and `%[NAME]#' to output group+ captures. A `%%' outputs `%'. See `ugrep --help format' and `man+ ugrep' section FORMAT for details. Context options -A, -B, -C and+ -y are ignored.++See [customized output with --format](#format) for details on the `FORMAT`+fields.++For option `-o`, the replacement is not automatically followed by a newline to+allow for more flexibility in replacements. To output a newline, use `%~` in+the `FORMAT` string.++Use option `-P` to use group captures and backreferences. Capturing groups in+regex patterns are parenthesized expressions `(pattern)` and the first is+referenced in `FORMAT` by `%1`, the second by `%2` and so on. Named captures+are of the form `(?<NAME>pattern)` and are referenced in `FORMAT` by+`%[NAME]#`.++To display pattern matches with their sequential match number using+`--replace='%m:%o'` where `%m` is the sequential match number and `%o` is the+pattern matched:++ ugrep --replace='%m:%o' pattern myfile.txt++Same, but passing the file through with option `-y`, while applying the+replacements to the output:++ ugrep -y --replace='%m:%o' pattern myfile.txt To extract table cells from an HTML file using Perl matching (`-P`) to support group captures with lazy quantifier `(.*?)`, and translate the matches to a comma-separated list with format `%,%1` (conditional comma and group capture): + ugrep -P -o '<td>(.*?)</td>' --replace='%,%1' index.html++Same, but using `--format='%,%1'` instead and we do not need `-o` (note that+`--replace` color-highlights matches shown on a terminal but `--format` does+not):+ ugrep -P '<td>(.*?)</td>' --format='%,%1' index.html -Same, but displaying the replaced matches line-by-line:+Same, but displaying the formatted matches line-by-line, with `--replace` or+with `--format`: - ugrep -P '<td>(.*?)</td>' --format='%1\n' index.html+ ugrep -P -o '<td>(.*?)</td>' --replace='%,%1' index.html+ ugrep -P '<td>(.*?)</td>' --format='%1%~' index.html To collect all `href` URLs from all HTML and PHP files down the working directory, then sort them:@@ -3366,8 +3454,8 @@ Same, but much easier by using the predefined `html/href` pattern: ugrep -R -thtml,php -P -f html/href --format='%1%~' | sort -u -Likewise, but in this case select `<script>` `src` URLs when referencing `http`-and `https` sites:+Same, but in this case select `<script>` `src` URLs when referencing `http` and+`https` sites: ugrep -R -thtml,php -P '<script.*src\h*=\h*.(https?:[^\x27"]+).' --format='%1%~' | sort -u @@ -3530,13 +3618,13 @@ Same, but sorted by time changed from most recent to oldest: ### Tips for advanced users When searching non-binary files only, the binary content check is disabled with-option `-a` to speed up displaying matches. For example, searching for line-with `int` in C++ source code:+option `-a` (`--text`) to speed up searching and displaying pattern matches.+For example, searching for lines with `int` in C++ source code: ugrep -r -a -Ocpp -w 'int' If a file has potentially many pattern matches, but each match is only one a-single line, then option `-u` can be used to speed up displaying matches:+single line, then option `-u` (`--ungroup`) can speed this up: ugrep -r -a -u -Opython -w 'def' @@ -3544,10 +3632,10 @@ Even greater speeds can be achieved with `--format` when searching files with many matches. For example, `--format='%O%~'` displays matching lines for each match on that line, while `--format='%o%~'` displays the matching part only. Note that the `--format` option does not check for binary matches, so the-output is "as is". To match text and binary, you can use `--format='%C%~'`-to display matches formatted as quoted C++ strings with escapes.-To display a line at most once (unless option `-u` is used), add the `%u`-(unique) field to the format string, e.g. `--format='%u%O%~'`.+output is always "as is". To match text and binary, you can use+`--format='%C%~'` to display matches formatted as quoted C++ strings with+escapes. To display a line at most once (unless option `-u` is used), add the+`%u` (unique) field to the format string, e.g. `--format='%u%O%~'`. For example, to match all words recursively in the working directory with line and column numbers, where `%n` is the line number, `%k` is the column number,@@ -3645,14 +3733,17 @@ in markdown: and defines complex regular expression patterns, then option -P (Perl matching) may improve performance (this omits POSIX DFA construction.) - For help, --help WHAT displays help on options related to WHAT.+ ugrep --help WHAT displays help on options related to WHAT; --help for-+ mat displays help on --format and --replace formatting; --help regex+ displays help on regular expression syntax and conventions; --help+ globs displays help on glob patterns. The following options are available: -A NUM, --after-context=NUM- Print NUM lines of trailing context after matching lines.+ Print NUM lines of trailing context after matching lines. Places a --group-separator between contiguous groups of matches.- See also options -B, -C, and -y.+ See also options -B, -C and -y. -a, --text Process a binary file as if it were text. This is equivalent to@@ -3661,108 +3752,107 @@ in markdown: if the terminal driver interprets some of it as commands. --and [[-e] PATTERN] ... -e PATTERN- Specify additional patterns to match. Patterns must be speci-- fied with -e. Each -e PATTERN following this option is consid-- ered an alternative pattern to match, i.e. each -e is inter-+ Specify additional patterns to match. Patterns must be speci-+ fied with -e. Each -e PATTERN following this option is consid-+ ered an alternative pattern to match, i.e. each -e is inter- preted as an OR pattern. For example, -e A -e B --and -e C -e D- matches lines with (`A' or `B') and (`C' or `D'). Note that- multiple -e PATTERN are alternations that bind more tightly- together than --and. Option --stats displays the search pat-+ matches lines with (`A' or `B') and (`C' or `D'). Note that+ multiple -e PATTERN are alternations that bind more tightly+ together than --and. Option --stats displays the search pat- terns applied. See also options --not, --andnot, --bool,- --files, and --lines.+ --files and --lines. --andnot [[-e] PATTERN] ...- Combines --and --not. See also options --and, --not, and- --bool.+ Combines --and --not. See also options --and, --not and --bool. -B NUM, --before-context=NUM- Print NUM lines of leading context before matching lines.+ Print NUM lines of leading context before matching lines. Places a --group-separator between contiguous groups of matches.- See also options -A, -C, and -y.+ See also options -A, -C and -y. -b, --byte-offset- The offset in bytes of a matched line is displayed in front of- the respective matched line. If -u is specified, displays the- offset for each pattern matched on the same line. Byte offsets- are exact for ASCII, UTF-8, and raw binary input. Otherwise,- the byte offset in the UTF-8 normalized input is displayed.+ The offset in bytes of a matched line is displayed in front of+ the respective matched line. If -u is specified, displays the+ offset for each pattern matched on the same line. Byte offsets+ are exact for ASCII, UTF-8 and raw binary input. Otherwise, the+ byte offset in the UTF-8 normalized input is displayed. --binary-files=TYPE- Controls searching and reporting pattern matches in binary- files. TYPE can be `binary', `without-match`, `text`, `hex`,- and `with-hex'. The default is `binary' to search binary files- and to report a match without displaying the match. `with-- out-match' ignores binary matches. `text' treats all binary- files as text, which might output binary garbage to the termi-- nal, which can have problematic consequences if the terminal- driver interprets some of it as commands. `hex' reports all- matches in hexadecimal. `with-hex' only reports binary matches- in hexadecimal, leaving text matches alone. A match is consid-- ered binary when matching a zero byte or invalid UTF. Short- options are -a, -I, -U, -W, and -X.+ Controls searching and reporting pattern matches in binary+ files. TYPE can be `binary', `without-match`, `text`, `hex` and+ `with-hex'. The default is `binary' to search binary files and+ to report a match without displaying the match. `without-match'+ ignores binary matches. `text' treats all binary files as text,+ which might output binary garbage to the terminal, which can+ have problematic consequences if the terminal driver interprets+ some of it as commands. `hex' reports all matches in hexadeci-+ mal. `with-hex' only reports binary matches in hexadecimal,+ leaving text matches alone. A match is considered binary when+ matching a zero byte or invalid UTF. Short options are -a, -I,+ -U, -W and -X. --bool, -%- Specifies Boolean query patterns. A Boolean query pattern is- composed of `AND', `OR', `NOT' operators and grouping with `('- `)'. Spacing between subpatterns is the same as `AND', `|' is- the same as `OR', and a `-' is the same as `NOT'. The `OR'- operator binds more tightly than `AND'. For example, --bool- 'A|B C|D' matches lines with (`A' or `B') and (`C' or `D'),- --bool 'A -B' matches lines with `A' and not `B'. Operators- `AND', `OR', `NOT' require proper spacing. For example, --bool- 'A OR B AND C OR D' matches lines with (`A' or `B') and (`C' or- `D'), --bool 'A AND NOT B' matches lines with `A' without `B'.- Quoted subpatterns are matched literally as strings. For exam-- ple, --bool 'A "AND"|"OR"' matches lines with `A' and also- either `AND' or `OR'. Parenthesis are used for grouping. For- example, --bool '(A B)|C' matches lines with `A' and `B', or- lines with `C'. Note that all subpatterns in a Boolean query- pattern are regular expressions, unless option -F is used.- Options -E, -F, -G, -P, and -Z can be combined with --bool to- match subpatterns as strings or regular expressions (-E is the- default.) This option does not apply to -f FILE patterns.- Option --stats displays the search patterns applied. See also- options --and, --andnot, --not, --files, and --lines.+ Specifies Boolean query patterns. A Boolean query pattern is+ composed of `AND', `OR', `NOT' operators and grouping with `('+ `)'. Spacing between subpatterns is the same as `AND', `|' is+ the same as `OR' and a `-' is the same as `NOT'. The `OR' oper-+ ator binds more tightly than `AND'. For example, --bool 'A|B+ C|D' matches lines with (`A' or `B') and (`C' or `D'), --bool 'A+ -B' matches lines with `A' and not `B'. Operators `AND', `OR',+ `NOT' require proper spacing. For example, --bool 'A OR B AND C+ OR D' matches lines with (`A' or `B') and (`C' or `D'), --bool+ 'A AND NOT B' matches lines with `A' without `B'. Quoted sub-+ patterns are matched literally as strings. For example, --bool+ 'A "AND"|"OR"' matches lines with `A' and also either `AND' or+ `OR'. Parenthesis are used for grouping. For example, --bool+ '(A B)|C' matches lines with `A' and `B', or lines with `C'.+ Note that all subpatterns in a Boolean query pattern are regular+ expressions, unless option -F is used. Options -E, -F, -G, -P+ and -Z can be combined with --bool to match subpatterns as+ strings or regular expressions (-E is the default.) This option+ does not apply to -f FILE patterns. Option --stats displays the+ search patterns applied. See also options --and, --andnot,+ --not, --files and --lines. --break Adds a line break between results from different files. -C NUM, --context=NUM Print NUM lines of leading and trailing context surrounding each- match. Places a --group-separator between contiguous groups of- matches. See also options -A, -B, and -y.+ match. Places a --group-separator between contiguous groups of+ matches. See also options -A, -B and -y. -c, --count- Only a count of selected lines is written to standard output.+ Only a count of selected lines is written to standard output. If -o or -u is specified, counts the number of patterns matched. If -v is specified, counts the number of non-matching lines. --color[=WHEN], --colour[=WHEN]- Mark up the matching text with the expression stored in the- GREP_COLOR or GREP_COLORS environment variable. WHEN can be+ Mark up the matching text with the expression stored in the+ GREP_COLOR or GREP_COLORS environment variable. WHEN can be `never', `always', or `auto', where `auto' marks up matches only when output on a terminal. The default is `auto'. --colors=COLORS, --colours=COLORS Use COLORS to mark up text. COLORS is a colon-separated list of- one or more parameters `sl=' (selected line), `cx=' (context- line), `mt=' (matched text), `ms=' (match selected), `mc='- (match context), `fn=' (file name), `ln=' (line number), `cn='+ one or more parameters `sl=' (selected line), `cx=' (context+ line), `mt=' (matched text), `ms=' (match selected), `mc='+ (match context), `fn=' (file name), `ln=' (line number), `cn=' (column number), `bn=' (byte offset), `se=' (separator). Param-- eter values are ANSI SGR color codes or `k' (black), `r' (red),- `g' (green), `y' (yellow), `b' (blue), `m' (magenta), `c'+ eter values are ANSI SGR color codes or `k' (black), `r' (red),+ `g' (green), `y' (yellow), `b' (blue), `m' (magenta), `c' (cyan), `w' (white). Upper case specifies background colors. A- `+' qualifies a color as bright. A foreground and a background- color may be combined with font properties `n' (normal), `f'+ `+' qualifies a color as bright. A foreground and a background+ color may be combined with font properties `n' (normal), `f' (faint), `h' (highlight), `i' (invert), `u' (underline). Param- eter `hl' enables file name hyperlinks. Parameter `rv' reverses- the `sl=' and `cx=' parameters with option -v. Selectively+ the `sl=' and `cx=' parameters with option -v. Selectively overrides GREP_COLORS. --config[=FILE], ---[FILE]- Use configuration FILE. The default FILE is `.ugrep'. The- working directory is checked first for FILE, then the home- directory. The options specified in the configuration FILE are+ Use configuration FILE. The default FILE is `.ugrep'. The+ working directory is checked first for FILE, then the home+ directory. The options specified in the configuration FILE are parsed first, followed by the remaining options specified on the command line. @@ -3771,148 +3861,150 @@ in markdown: --cpp Output file matches in C++. See also options --format and -u. - --csv Output file matches in CSV. If -H, -n, -k, or -b is specified,+ --csv Output file matches in CSV. If -H, -n, -k, or -b is specified, additional values are output. See also options --format and -u. -D ACTION, --devices=ACTION- If an input file is a device, FIFO or socket, use ACTION to- process it. By default, ACTION is `skip', which means that+ If an input file is a device, FIFO or socket, use ACTION to+ process it. By default, ACTION is `skip', which means that devices are silently skipped. If ACTION is `read', devices read just as if they were ordinary files. -d ACTION, --directories=ACTION- If an input file is a directory, use ACTION to process it. By- default, ACTION is `skip', i.e., silently skip directories+ If an input file is a directory, use ACTION to process it. By+ default, ACTION is `skip', i.e., silently skip directories unless specified on the command line. If ACTION is `read', warn- when directories are read as input. If ACTION is `recurse',+ when directories are read as input. If ACTION is `recurse', read all files under each directory, recursively, following sym-- bolic links only if they are on the command line. This is- equivalent to the -r option. If ACTION is `derefer-+ bolic links only if they are on the command line. This is+ equivalent to the -r option. If ACTION is `derefer- ence-recurse', read all files under each directory, recursively,- following symbolic links. This is equivalent to the -R option.+ following symbolic links. This is equivalent to the -R option. --depth=[MIN,][MAX], -1, -2 ... -9, --10, --11 ...- Restrict recursive searches from MIN to MAX directory levels- deep, where -1 (--depth=1) searches the specified path without- recursing into subdirectories. Note that -3 -5, -3-5, or -35- searches 3 to 5 levels deep. Enables -R if -R or -r is not+ Restrict recursive searches from MIN to MAX directory levels+ deep, where -1 (--depth=1) searches the specified path without+ recursing into subdirectories. Note that -3 -5, -3-5, or -35+ searches 3 to 5 levels deep. Enables -R if -R or -r is not specified. --dotall- Dot `.' in regular expressions matches anything, including new-- line. Note that `.*' matches all input and should not be used.+ Dot `.' in regular expressions matches anything, including new-+ line. Note that `.*' matches all input and should not be used. -E, --extended-regexp- Interpret patterns as extended regular expressions (EREs). This+ Interpret patterns as extended regular expressions (EREs). This is the default. -e PATTERN, --regexp=PATTERN- Specify a PATTERN used during the search of the input: an input- line is selected if it matches any of the specified patterns.+ Specify a PATTERN used during the search of the input: an input+ line is selected if it matches any of the specified patterns. Note that longer patterns take precedence over shorter patterns.- This option is most useful when multiple -e options are used to- specify multiple patterns, when a pattern begins with a dash- (`-'), to specify a pattern after option -f or after the FILE+ This option is most useful when multiple -e options are used to+ specify multiple patterns, when a pattern begins with a dash+ (`-'), to specify a pattern after option -f or after the FILE arguments. --encoding=ENCODING- The encoding format of the input, where ENCODING can be:- `binary', `ASCII', `UTF-8', `UTF-16', `UTF-16BE', `UTF-16LE',- `UTF-32', `UTF-32BE', `UTF-32LE', `LATIN1', `ISO-8859-1',+ The encoding format of the input, where ENCODING can be:+ `binary', `ASCII', `UTF-8', `UTF-16', `UTF-16BE', `UTF-16LE',+ `UTF-32', `UTF-32BE', `UTF-32LE', `LATIN1', `ISO-8859-1', `ISO-8859-2', `ISO-8859-3', `ISO-8859-4', `ISO-8859-5', `ISO-8859-6', `ISO-8859-7', `ISO-8859-8', `ISO-8859-9',- `ISO-8859-10', `ISO-8859-11', `ISO-8859-13', `ISO-8859-14',- `ISO-8859-15', `ISO-8859-16', `MAC', `MACROMAN', `EBCDIC',- `CP437', `CP850', `CP858', `CP1250', `CP1251', `CP1252',+ `ISO-8859-10', `ISO-8859-11', `ISO-8859-13', `ISO-8859-14',+ `ISO-8859-15', `ISO-8859-16', `MAC', `MACROMAN', `EBCDIC',+ `CP437', `CP850', `CP858', `CP1250', `CP1251', `CP1252', `CP1253', `CP1254', `CP1255', `CP1256', `CP1257', `CP1258', `KOI8-R', `KOI8-U', `KOI8-RU'. --exclude=GLOB Skip files whose name matches GLOB using wildcard matching, same- as -g ^GLOB. GLOB can use **, *, ?, and [...] as wildcards, and- \ to quote a wildcard or backslash character literally. When- GLOB contains a `/', full pathnames are matched. Otherwise- basenames are matched. When GLOB ends with a `/', directories- are excluded as if --exclude-dir is specified. Otherwise files- are excluded. Note that --exclude patterns take priority over- --include patterns. GLOB should be quoted to prevent shell+ as -g ^GLOB. GLOB can use **, *, ?, and [...] as wildcards and+ \ to quote a wildcard or backslash character literally. When+ GLOB contains a `/', full pathnames are matched. Otherwise+ basenames are matched. When GLOB ends with a `/', directories+ are excluded as if --exclude-dir is specified. Otherwise files+ are excluded. Note that --exclude patterns take priority over+ --include patterns. GLOB should be quoted to prevent shell globbing. This option may be repeated. --exclude-dir=GLOB- Exclude directories whose name matches GLOB from recursive- searches, same as -g ^GLOB/. GLOB can use **, *, ?, and [...]- as wildcards, and \ to quote a wildcard or backslash character+ Exclude directories whose name matches GLOB from recursive+ searches, same as -g ^GLOB/. GLOB can use **, *, ?, and [...]+ as wildcards and \ to quote a wildcard or backslash character literally. When GLOB contains a `/', full pathnames are- matched. Otherwise basenames are matched. Note that- --exclude-dir patterns take priority over --include-dir pat-- terns. GLOB should be quoted to prevent shell globbing. This+ matched. Otherwise basenames are matched. Note that+ --exclude-dir patterns take priority over --include-dir pat-+ terns. GLOB should be quoted to prevent shell globbing. This option may be repeated. --exclude-from=FILE- Read the globs from FILE and skip files and directories whose- name matches one or more globs. A glob can use **, *, ?, and- [...] as wildcards, and \ to quote a wildcard or backslash char-+ Read the globs from FILE and skip files and directories whose+ name matches one or more globs. A glob can use **, *, ?, and+ [...] as wildcards and \ to quote a wildcard or backslash char- acter literally. When a glob contains a `/', full pathnames are- matched. Otherwise basenames are matched. When a glob ends- with a `/', directories are excluded as if --exclude-dir is- specified. Otherwise files are excluded. A glob starting with- a `!' overrides previously-specified exclusions by including- matching files. Lines starting with a `#' and empty lines in- FILE are ignored. When FILE is a `-', standard input is read.+ matched. Otherwise basenames are matched. When a glob ends+ with a `/', directories are excluded as if --exclude-dir is+ specified. Otherwise files are excluded. A glob starting with+ a `!' overrides previously-specified exclusions by including+ matching files. Lines starting with a `#' and empty lines in+ FILE are ignored. When FILE is a `-', standard input is read. This option may be repeated. --exclude-fs=MOUNTS- Exclude file systems specified by MOUNTS from recursive- searches, MOUNTS is a comma-separated list of mount points or- pathnames of directories on file systems. Note that- --exclude-fs mounts take priority over --include-fs mounts.+ Exclude file systems specified by MOUNTS from recursive+ searches, MOUNTS is a comma-separated list of mount points or+ pathnames of directories on file systems. Note that+ --exclude-fs mounts take priority over --include-fs mounts. This option may be repeated. -F, --fixed-strings- Interpret pattern as a set of fixed strings, separated by new-- lines, any of which is to be matched. This makes ugrep behave- as fgrep. If a PATTERN is specified, or -e PATTERN or -N PAT-- TERN, then this option has no effect on -f FILE patterns to- allow -f FILE patterns to narrow or widen the scope of the PAT-+ Interpret pattern as a set of fixed strings, separated by new-+ lines, any of which is to be matched. This makes ugrep behave+ as fgrep. If a PATTERN is specified, or -e PATTERN or -N PAT-+ TERN, then this option has no effect on -f FILE patterns to+ allow -f FILE patterns to narrow or widen the scope of the PAT- TERN search. -f FILE, --file=FILE- Read newline-separated patterns from FILE. White space in pat-+ Read newline-separated patterns from FILE. White space in pat- terns is significant. Empty lines in FILE are ignored. If FILE- does not exist, the GREP_PATH environment variable is used as- path to FILE. If that fails, looks for FILE in- /usr/local/share/ugrep/patterns. When FILE is a `-', standard+ does not exist, the GREP_PATH environment variable is used as+ path to FILE. If that fails, looks for FILE in+ /usr/local/share/ugrep/patterns. When FILE is a `-', standard input is read. Empty files contain no patterns; thus nothing is matched. This option may be repeated. --filter=COMMANDS Filter files through the specified COMMANDS first before search- ing. COMMANDS is a comma-separated list of `exts:command- [option ...]', where `exts' is a comma-separated list of file-- name extensions and `command' is a filter utility. The filter- utility should read from standard input and write to standard+ [option ...]', where `exts' is a comma-separated list of file-+ name extensions and `command' is a filter utility. The filter+ utility should read from standard input and write to standard output. Files matching one of `exts' are filtered. When `exts' is `*', files with non-matching extensions are filtered. One or- more `option' separated by spacing may be specified, which are- passed verbatim to the command. A `%' as `option' expands into- the pathname to search. For example, --filter='pdf:pdftotext %- -' searches PDF files. The `%' expands into a `-' when search-- ing standard input. Option --label=.ext may be used to specify+ more `option' separated by spacing may be specified, which are+ passed verbatim to the command. A `%' as `option' expands into+ the pathname to search. For example, --filter='pdf:pdftotext %+ -' searches PDF files. The `%' expands into a `-' when search-+ ing standard input. Option --label=.ext may be used to specify extension `ext' when searching standard input. --filter-magic-label=[+]LABEL:MAGIC- Associate LABEL with files whose signature "magic bytes" match- the MAGIC regex pattern. Only files that have no filename- extension are labeled, unless +LABEL is specified. When LABEL- matches an extension specified in --filter=COMMANDS, the corre-+ Associate LABEL with files whose signature "magic bytes" match+ the MAGIC regex pattern. Only files that have no filename+ extension are labeled, unless +LABEL is specified. When LABEL+ matches an extension specified in --filter=COMMANDS, the corre- sponding command is invoked. This option may be repeated. --format=FORMAT- Output FORMAT-formatted matches. For example --for-- mat='%f:%n:%O%~' outputs matching lines `%O' with filename `%f`- and line number `%n' followed by a newline `%~'. Context- options -A, -B, -C, and -y are ignored. See `man ugrep' section- FORMAT.+ Output FORMAT-formatted matches. For example --for-+ mat='%f:%n:%O%~' outputs matching lines `%O' with filename `%f`+ and line number `%n' followed by a newline `%~'. When option -P+ is used, FORMAT may include `%1' to `%9', `%[NUM]#' and+ `%[NAME]#' to output group captures. A `%%' outputs `%'. See+ `ugrep --help format' and `man ugrep' section FORMAT for+ details. Context options -A, -B, -C and -y are ignored. --free-space Spacing (blanks and tabs) in regular expressions are ignored.@@ -3922,125 +4014,129 @@ in markdown: behave as traditional grep. -g GLOBS, --glob=GLOBS- Search only files whose name matches the specified comma-sepa-+ Search only files whose name matches the specified comma-sepa- rated list of GLOBS, same as --include='glob' for each `glob' in- GLOBS. When a `glob' is preceded by a `!' or a `^', skip files- whose name matches `glob', same as --exclude='glob'. When- `glob' contains a `/', full pathnames are matched. Otherwise+ GLOBS. When a `glob' is preceded by a `!' or a `^', skip files+ whose name matches `glob', same as --exclude='glob'. When+ `glob' contains a `/', full pathnames are matched. Otherwise basenames are matched. When `glob' ends with a `/', directories- are matched, same as --include-dir='glob' and- --exclude-dir='glob'. A leading `/' matches the working direc-- tory. This option may be repeated and may be combined with+ are matched, same as --include-dir='glob' and+ --exclude-dir='glob'. A leading `/' matches the working direc-+ tory. This option may be repeated and may be combined with options -M, -O and -t to expand the recursive search. --group-separator[=SEP]- Use SEP as a group separator for context options -A, -B, and -C.+ Use SEP as a group separator for context options -A, -B and -C. The default is a double hyphen (`--'). -H, --with-filename- Always print the filename with output lines. This is the+ Always print the filename with output lines. This is the default when there is more than one file to search. -h, --no-filename- Never print filenames with output lines. This is the default- when there is only one file (or only standard input) to search.+ Never print filenames with output lines. This is the default+ when there is only one file (or only standard input) to search. --heading, -+ Group matches per file. Adds a heading and a line break between results from different files. --help [WHAT], -? [WHAT]- Display a help message, specifically on WHAT when specified.+ Display a help message, specifically on WHAT when specified. In+ addition, `--help format' displays an overview of FORMAT fields,+ `--help regex' displays an overview of regular expressions and+ `--help globs' displays an overview of glob syntax and conven-+ tions. --hexdump=[1-8][a][b][c][h]- Output matches in 1 to 8 columns of 8 hexadecimal octets. The- default is 2 columns or 16 octets per line. Option `a' outputs- a `*' for all hex lines that are identical to the previous hex- line, `b' removes all space breaks, `c' removes the character- column and `h' removes hex spacing. Enables -X if -W or -X is+ Output matches in 1 to 8 columns of 8 hexadecimal octets. The+ default is 2 columns or 16 octets per line. Option `a' outputs+ a `*' for all hex lines that are identical to the previous hex+ line, `b' removes all space breaks, `c' removes the character+ column and `h' removes hex spacing. Enables -X if -W or -X is not specified. --hidden, -. Search hidden files and directories. --hyperlink- Hyperlinks are enabled for file names when colors are enabled.+ Hyperlinks are enabled for file names when colors are enabled. Same as --colors=hl. -I, --ignore-binary- Ignore matches in binary files. This option is equivalent to+ Ignore matches in binary files. This option is equivalent to the --binary-files=without-match option. -i, --ignore-case- Perform case insensitive matching. By default, ugrep is case- sensitive. By default, this option applies to ASCII letters+ Perform case insensitive matching. By default, ugrep is case+ sensitive. By default, this option applies to ASCII letters only. Use options -P and -i for Unicode case insensitive match- ing. --ignore-files[=FILE]- Ignore files and directories matching the globs in each FILE- that is encountered in recursive searches. The default FILE is- `.gitignore'. Matching files and directories located in the- directory of a FILE's location and in directories below are- ignored by temporarily overriding the --exclude and- --exclude-dir globs, as if --exclude-from=FILE is locally- enforced. Globbing is the same as --exclude-from=FILE and sup-- ports gitignore syntax, but directories are not automatically+ Ignore files and directories matching the globs in each FILE+ that is encountered in recursive searches. The default FILE is+ `.gitignore'. Matching files and directories located in the+ directory of a FILE's location and in directories below are+ ignored by temporarily overriding the --exclude and+ --exclude-dir globs, as if --exclude-from=FILE is locally+ enforced. Globbing is the same as --exclude-from=FILE and sup-+ ports gitignore syntax, but directories are not automatically excluded from searches (use a glob ending with a `/' to identify- directories to ignore, same as git). Files and directories- explicitly specified as command line arguments are never+ directories to ignore, same as git). Files and directories+ explicitly specified as command line arguments are never ignored. This option may be repeated. --include=GLOB- Search only files whose name matches GLOB using wildcard match-+ Search only files whose name matches GLOB using wildcard match- ing, same as -g GLOB. GLOB can use **, *, ?, and [...] as wild-- cards, and \ to quote a wildcard or backslash character liter-- ally. When GLOB contains a `/', full pathnames are matched.- Otherwise basenames are matched. When GLOB ends with a `/',+ cards and \ to quote a wildcard or backslash character liter-+ ally. When GLOB contains a `/', full pathnames are matched.+ Otherwise basenames are matched. When GLOB ends with a `/', directories are included as if --include-dir is specified. Oth-- erwise files are included. Note that --exclude patterns take+ erwise files are included. Note that --exclude patterns take priority over --include patterns. GLOB should be quoted to pre- vent shell globbing. This option may be repeated. --include-dir=GLOB- Only directories whose name matches GLOB are included in recur-- sive searches, same as -g GLOB/. GLOB can use **, *, ?, and- [...] as wildcards, and \ to quote a wildcard or backslash char-- acter literally. When GLOB contains a `/', full pathnames are- matched. Otherwise basenames are matched. Note that- --exclude-dir patterns take priority over --include-dir pat-- terns. GLOB should be quoted to prevent shell globbing. This+ Only directories whose name matches GLOB are included in recur-+ sive searches, same as -g GLOB/. GLOB can use **, *, ?, and+ [...] as wildcards and \ to quote a wildcard or backslash char-+ acter literally. When GLOB contains a `/', full pathnames are+ matched. Otherwise basenames are matched. Note that+ --exclude-dir patterns take priority over --include-dir pat-+ terns. GLOB should be quoted to prevent shell globbing. This option may be repeated. --include-from=FILE- Read the globs from FILE and search only files and directories- whose name matches one or more globs. A glob can use **, *, ?,- and [...] as wildcards, and \ to quote a wildcard or backslash+ Read the globs from FILE and search only files and directories+ whose name matches one or more globs. A glob can use **, *, ?,+ and [...] as wildcards and \ to quote a wildcard or backslash character literally. When a glob contains a `/', full pathnames are matched. Otherwise basenames are matched. When a glob ends- with a `/', directories are included as if --include-dir is- specified. Otherwise files are included. A glob starting with- a `!' overrides previously-specified inclusions by excluding- matching files. Lines starting with a `#' and empty lines in- FILE are ignored. When FILE is a `-', standard input is read.+ with a `/', directories are included as if --include-dir is+ specified. Otherwise files are included. A glob starting with+ a `!' overrides previously-specified inclusions by excluding+ matching files. Lines starting with a `#' and empty lines in+ FILE are ignored. When FILE is a `-', standard input is read. This option may be repeated. --include-fs=MOUNTS- Only file systems specified by MOUNTS are included in recursive- searches. MOUNTS is a comma-separated list of mount points or- pathnames of directories on file systems. --include-fs=.- restricts recursive searches to the file system of the working- directory only. Note that --exclude-fs mounts take priority+ Only file systems specified by MOUNTS are included in recursive+ searches. MOUNTS is a comma-separated list of mount points or+ pathnames of directories on file systems. --include-fs=.+ restricts recursive searches to the file system of the working+ directory only. Note that --exclude-fs mounts take priority over --include-fs mounts. This option may be repeated. -J NUM, --jobs=NUM- Specifies the number of threads spawned to search files. By- default an optimum number of threads is spawned to search files- simultaneously. -J1 disables threading: files are searched in+ Specifies the number of threads spawned to search files. By+ default an optimum number of threads is spawned to search files+ simultaneously. -J1 disables threading: files are searched in the same order as specified. -j, --smart-case- Perform case insensitive matching like option -i, unless a pat-+ Perform case insensitive matching like option -i, unless a pat- tern is specified with a literal ASCII upper case letter. --json Output file matches in JSON. If -H, -n, -k, or -b is specified,@@ -4050,26 +4146,26 @@ in markdown: Start searching at line FIRST, stop at line LAST when specified. -k, --column-number- The column number of a matched pattern is displayed in front of- the respective matched line, starting at column 1. Tabs are+ The column number of a matched pattern is displayed in front of+ the respective matched line, starting at column 1. Tabs are expanded when columns are counted, see also option --tabs. -L, --files-without-match- Only the names of files not containing selected lines are writ-- ten to standard output. Pathnames are listed once per file+ Only the names of files not containing selected lines are writ-+ ten to standard output. Pathnames are listed once per file searched. If the standard input is searched, the string ``(standard input)'' is written. -l, --files-with-matches Only the names of files containing selected lines are written to- standard output. ugrep will only search a file until a match- has been found, making searches potentially less expensive.- Pathnames are listed once per file searched. If the standard+ standard output. ugrep will only search a file until a match+ has been found, making searches potentially less expensive.+ Pathnames are listed once per file searched. If the standard input is searched, the string ``(standard input)'' is written. --label=LABEL- Displays the LABEL value when input is read from standard input- where a file name would normally be printed in the output. As-+ Displays the LABEL value when input is read from standard input+ where a file name would normally be printed in the output. As- sociates a filename extension with standard input when LABEL has a suffix. The default value is `(standard input)'. @@ -4077,17 +4173,17 @@ in markdown: Force output to be line buffered instead of block buffered. --lines- Apply Boolean queries to match lines, the opposite of --files.- This is the default Boolean query mode to match specific lines.+ Apply Boolean queries to match lines, the opposite of --files.+ This is the default Boolean query mode to match specific lines. -M MAGIC, --file-magic=MAGIC- Only files matching the signature pattern MAGIC are searched.- The signature "magic bytes" at the start of a file are compared- to the MAGIC regex pattern. When matching, the file will be- searched. When MAGIC is preceded by a `!' or a `^', skip files+ Only files matching the signature pattern MAGIC are searched.+ The signature "magic bytes" at the start of a file are compared+ to the MAGIC regex pattern. When matching, the file will be+ searched. When MAGIC is preceded by a `!' or a `^', skip files with matching MAGIC signatures. This option may be repeated and- may be combined with options -O and -t to expand the search.- Every file on the search path is read, making searches poten-+ may be combined with options -O and -t to expand the search.+ Every file on the search path is read, making searches poten- tially more expensive. -m NUM, --max-count=NUM@@ -4097,108 +4193,108 @@ in markdown: Match all input. Same as specifying an empty pattern to search. --max-files=NUM- Restrict the number of files matched to NUM. Note that --sort- or -J1 may be specified to produce replicable results. If+ Restrict the number of files matched to NUM. Note that --sort+ or -J1 may be specified to produce replicable results. If --sort is specified, the number of threads spawned is limited to NUM. --mmap[=MAX]- Use memory maps to search files. By default, memory maps are- used under certain conditions to improve performance. When MAX+ Use memory maps to search files. By default, memory maps are+ used under certain conditions to improve performance. When MAX is specified, use up to MAX mmap memory per thread. -N PATTERN, --neg-regexp=PATTERN- Specify a negative PATTERN used during the search of the input:- an input line is selected only if it matches any of the speci-- fied patterns unless a subpattern of PATTERN. Same as -e- (?^PATTERN). Negative PATTERN matches are essentially removed- before any other patterns are matched. Note that longer pat-+ Specify a negative PATTERN used during the search of the input:+ an input line is selected only if it matches any of the speci-+ fied patterns unless a subpattern of PATTERN. Same as -e+ (?^PATTERN). Negative PATTERN matches are essentially removed+ before any other patterns are matched. Note that longer pat- terns take precedence over shorter patterns. This option may be repeated. -n, --line-number- Each output line is preceded by its relative line number in the- file, starting at line 1. The line number counter is reset for+ Each output line is preceded by its relative line number in the+ file, starting at line 1. The line number counter is reset for each file processed. --no-group-separator- Removes the group separator line from the output for context- options -A, -B, and -C.+ Removes the group separator line from the output for context+ options -A, -B and -C. --not [-e] PATTERN- Specifies that PATTERN should not match. Note that -e A --not- -e B matches lines with `A' or lines without a `B'. To match- lines with `A' that have no `B', specify -e A --andnot -e B.- Option --stats displays the search patterns applied. See also- options --and, --andnot, --bool, --files, and --lines.+ Specifies that PATTERN should not match. Note that -e A --not+ -e B matches lines with `A' or lines without a `B'. To match+ lines with `A' that have no `B', specify -e A --andnot -e B.+ Option --stats displays the search patterns applied. See also+ options --and, --andnot, --bool, --files and --lines. -O EXTENSIONS, --file-extension=EXTENSIONS- Search only files whose filename extensions match the specified- comma-separated list of EXTENSIONS, same as --include='*.ext'- for each `ext' in EXTENSIONS. When an `ext' is preceded by a- `!' or a `^', skip files whose filename extensions matches- `ext', same as --exclude='*.ext'. This option may be repeated- and may be combined with options -g, -M and -t to expand the+ Search only files whose filename extensions match the specified+ comma-separated list of EXTENSIONS, same as --include='*.ext'+ for each `ext' in EXTENSIONS. When an `ext' is preceded by a+ `!' or a `^', skip files whose filename extensions matches+ `ext', same as --exclude='*.ext'. This option may be repeated+ and may be combined with options -g, -M and -t to expand the recursive search. -o, --only-matching- Print only the matching part of lines. When multiple lines- match, the line numbers with option -n are displayed using `|'- as the field separator for each additional line matched by the- pattern. If -u is specified, ungroups multiple matches on the- same line. This option cannot be combined with options -A, -B,- -C, -v, and -y.+ Print only the matching part of lines. When multiple lines+ match, the line numbers with option -n are displayed using `|'+ as the field separator for each additional line matched by the+ pattern. If -u is specified, ungroups multiple matches on the+ same line. This option cannot be combined with options -A, -B,+ -C, -v and -y. --only-line-number The line number of the matching line in the file is output with-- out displaying the match. The line number counter is reset for+ out displaying the match. The line number counter is reset for each file processed. --files- Apply Boolean queries to match files, the opposite of --lines.- A file matches if all Boolean conditions are satisfied by the+ Apply Boolean queries to match files, the opposite of --lines.+ A file matches if all Boolean conditions are satisfied by the lines matched in the file. For example, --files -e A --and -e B- -e C --andnot -e D matches a file if some lines match `A' and- some lines match (`B' or `C') and no line in the file matches- `D'. May also be specified as --files --bool 'A B|C -D'.- Option -v cannot be specified with --files. See also options+ -e C --andnot -e D matches a file if some lines match `A' and+ some lines match (`B' or `C') and no line in the file matches+ `D'. May also be specified as --files --bool 'A B|C -D'.+ Option -v cannot be specified with --files. See also options --and, --andnot, --not, --bool and --lines. -P, --perl-regexp- Interpret PATTERN as a Perl regular expression using PCRE2.- Note that Perl pattern matching differs from the default grep+ Interpret PATTERN as a Perl regular expression using PCRE2.+ Note that Perl pattern matching differs from the default grep POSIX pattern matching. -p, --no-dereference- If -R or -r is specified, no symbolic links are followed, even+ If -R or -r is specified, no symbolic links are followed, even when they are specified on the command line. --pager[=COMMAND]- When output is sent to the terminal, uses COMMAND to page- through the output. The default COMMAND is `less -R'. Enables+ When output is sent to the terminal, uses COMMAND to page+ through the output. The default COMMAND is `less -R'. Enables --heading and --line-buffered. --pretty- When output is sent to a terminal, enables --color, --heading,+ When output is sent to a terminal, enables --color, --heading, -n, --sort and -T when not explicitly disabled or set. -Q[DELAY], --query[=DELAY]- Query mode: user interface to perform interactive searches.- This mode requires an ANSI capable terminal. An optional DELAY- argument may be specified to reduce or increase the response+ Query mode: user interface to perform interactive searches.+ This mode requires an ANSI capable terminal. An optional DELAY+ argument may be specified to reduce or increase the response time to execute searches after the last key press, in increments- of 100ms, where the default is 5 (0.5s delay). No whitespace- may be given between -Q and its argument DELAY. Initial pat-- terns may be specified with -e PATTERN, i.e. a PATTERN argument+ of 100ms, where the default is 5 (0.5s delay). No whitespace+ may be given between -Q and its argument DELAY. Initial pat-+ terns may be specified with -e PATTERN, i.e. a PATTERN argument requires option -e. Press F1 or CTRL-Z to view the help screen.- Press F2 or CTRL-Y to invoke a command to view or edit the file- shown at the top of the screen. The command can be specified+ Press F2 or CTRL-Y to invoke a command to view or edit the file+ shown at the top of the screen. The command can be specified with option --view, or defaults to environment variable PAGER if- defined, or EDITOR. Press Tab and Shift-Tab to navigate direc-- tories and to select a file to search. Press Enter to select+ defined, or EDITOR. Press Tab and Shift-Tab to navigate direc-+ tories and to select a file to search. Press Enter to select lines to output. Press ALT-l for option -l to list files, ALT-n- for -n, etc. Non-option commands include ALT-] to increase- fuzziness and ALT-} to increase context. Enables --heading.+ for -n, etc. Non-option commands include ALT-] to increase+ fuzziness and ALT-} to increase context. Enables --heading. See also options --confirm and --view. -q, --quiet, --silent@@ -4206,17 +4302,25 @@ in markdown: match has been found. -R, --dereference-recursive- Recursively read all files under each directory. Follow all- symbolic links, unlike -r. When -J1 is specified, files are+ Recursively read all files under each directory. Follow all+ symbolic links, unlike -r. When -J1 is specified, files are searched in the same order as specified. Note that when no FILE- arguments are specified and input is read from a terminal,+ arguments are specified and input is read from a terminal, recursive searches are performed as if -R is specified. -r, --recursive- Recursively read all files under each directory, following sym-- bolic links only if they are on the command line. When -J1 is+ Recursively read all files under each directory, following sym-+ bolic links only if they are on the command line. When -J1 is specified, files are searched in the same order as specified. + --replace=FORMAT+ Replace matching patterns in the output by the specified FORMAT+ with `%' fields. When option -P is used, FORMAT may include+ `%1' to `%9', `%[NUM]#' and `%[NAME]#' to output group captures.+ A `%%' outputs `%' and `%~' outputs a newline. See option+ --format, `ugrep --help format' and `man ugrep' section FORMAT+ for details.+ -S, --dereference If -r is specified, all symbolic links are followed, like -R. The default is not to follow symbolic links.@@ -4231,7 +4335,7 @@ in markdown: --separator[=SEP] Use SEP as field separator between file name, line number, col-- umn number, byte offset, and the matched line. The default is a+ umn number, byte offset and the matched line. The default is a colon (`:'). --sort[=KEY]@@ -4240,7 +4344,7 @@ in markdown: `best' to sort by best match with option -Z (sort by best match requires two passes over the input files), `size' to sort by file size, `used' to sort by last access time, `changed' to sort- by last modification time, and `created' to sort by creation+ by last modification time and `created' to sort by creation time. Sorting is reversed with `rname', `rbest', `rsize', `rused', `rchanged', or `rcreated'. Archive contents are not sorted. Subdirectories are sorted and displayed after matching@@ -4250,11 +4354,11 @@ in markdown: --stats Output statistics on the number of files and directories- searched, and the inclusion and exclusion constraints applied.+ searched and the inclusion and exclusion constraints applied. -T, --initial-tab Add a tab space to separate the file name, line number, column- number, and byte offset with the matched line.+ number and byte offset with the matched line. -t TYPES, --file-type=TYPES Search only files associated with TYPES, a comma-separated list@@ -4301,7 +4405,7 @@ in markdown: using `+' as the field separator. -V, --version- Display version information and exit.+ Display version with linked libraries and exit. -v, --invert-match Selected lines are those not matching any of the specified pat-@@ -4319,13 +4423,13 @@ in markdown: -w, --word-regexp The PATTERN is searched for as a word, such that the matching text is preceded by a non-word character and is followed by a- non-word character. Word characters are letters, digits, and- the underscore. With option -P, word characters are Unicode- letters, digits, and underscore. This option has no effect if- -x is also specified. If a PATTERN is specified, or -e PATTERN- or -N PATTERN, then this option has no effect on -f FILE pat-- terns to allow -f FILE patterns to narrow or widen the scope of- the PATTERN search.+ non-word character. Word characters are letters, digits and the+ underscore. With option -P, word characters are Unicode let-+ ters, digits and underscore. This option has no effect if -x is+ also specified. If a PATTERN is specified, or -e PATTERN or -N+ PATTERN, then this option has no effect on -f FILE patterns to+ allow -f FILE patterns to narrow or widen the scope of the PAT-+ TERN search. -X, --hex Output matches in hexadecimal. This option is equivalent to the@@ -4349,63 +4453,63 @@ in markdown: -y, --any-line Any line is output (passthru). Non-matching lines are output as- context with a `-' separator. See also options -A, -B, and -C.+ context with a `-' separator. See also options -A, -B and -C. -Z[[+-~]MAX], --fuzzy[=[+-~]MAX] Fuzzy mode: report approximate pattern matches within MAX- errors. By default, MAX is 1: one deletion, insertion or sub-- stitution is allowed. When `+' and/or `-' precede MAX, only+ errors. By default, MAX is 1: one deletion, insertion or sub-+ stitution is allowed. When `+' and/or `-' precede MAX, only insertions and/or deletions are allowed, respectively. When `~'- precedes MAX, substitution counts as one error. For example,- -Z+~3 allows up to three insertions or substitutions, but no- deletions. The first character of an approximate match always- matches the begin of a pattern. Option --sort=best orders- matching files by best match. No whitespace may be given+ precedes MAX, substitution counts as one error. For example,+ -Z+~3 allows up to three insertions or substitutions, but no+ deletions. The first character of an approximate match always+ matches the begin of a pattern. Option --sort=best orders+ matching files by best match. No whitespace may be given between -Z and its argument. -z, --decompress- Decompress files to search, when compressed. Archives (.cpio,- .pax, .tar and .zip) and compressed archives (e.g. .taz, .tgz,- .tpz, .tbz, .tbz2, .tb2, .tz2, .tlz, .txz, .tzst) are searched- and matching pathnames of files in archives are output in- braces. If -g, -O, -M, or -t is specified, searches files- stored in archives whose filenames match globs, match filename- extensions, match file signature magic bytes, or match file+ Decompress files to search, when compressed. Archives (.cpio,+ .pax, .tar and .zip) and compressed archives (e.g. .taz, .tgz,+ .tpz, .tbz, .tbz2, .tb2, .tz2, .tlz, .txz, .tzst) are searched+ and matching pathnames of files in archives are output in+ braces. If -g, -O, -M, or -t is specified, searches files+ stored in archives whose filenames match globs, match filename+ extensions, match file signature magic bytes, or match file types, respectively. Supported compression formats: gzip (.gz),- compress (.Z), zip, bzip2 (requires suffix .bz, .bz2, .bzip2,- .tbz, .tbz2, .tb2, .tz2), lzma and xz (requires suffix .lzma,- .tlz, .xz, .txz), lz4 (requires suffix .lz4), zstd (requires+ compress (.Z), zip, bzip2 (requires suffix .bz, .bz2, .bzip2,+ .tbz, .tbz2, .tb2, .tz2), lzma and xz (requires suffix .lzma,+ .tlz, .xz, .txz), lz4 (requires suffix .lz4), zstd (requires suffix .zst, .zstd, .tzst). --zmax=NUM- When used with option -z (--decompress), searches the contents- of compressed files and archives stored within archives for up- to NUM recursive levels deep. The default --zmax=1 only permits- searching uncompressed files stored in cpio, pax, tar and zip- archives; compressed files and archives are detected as binary- files and are effectively ignored. Specify --zmax=2 to search- compressed files and archives stored in cpio, pax, tar and zip+ When used with option -z (--decompress), searches the contents+ of compressed files and archives stored within archives by up to+ NUM recursive expansions. The default --zmax=1 only permits+ searching uncompressed files stored in cpio, pax, tar and zip+ archives; compressed files and archives are detected as binary+ files and are effectively ignored. Specify --zmax=2 to search+ compressed files and archives stored in cpio, pax, tar and zip archives. NUM may range from 1 to 99 for up to 99 decompression- and de-archiving steps. Larger NUM values gradually degrade- performance.+ and de-archiving steps. Increasing NUM values gradually+ degrades performance. -0, --null- Prints a zero-byte (NUL) after the file name. This option can- be used with commands such as `find -print0' and `xargs -0' to+ Prints a zero-byte (NUL) after the file name. This option can+ be used with commands such as `find -print0' and `xargs -0' to process arbitrary file names. - A `--' signals the end of options; the rest of the parameters are FILE+ A `--' signals the end of options; the rest of the parameters are FILE arguments, allowing filenames to begin with a `-' character. Long options may start with `--no-' to disable, when applicable. - The regular expression pattern syntax is an extended form of the POSIX+ The regular expression pattern syntax is an extended form of the POSIX ERE syntax. For an overview of the syntax see README.md or visit: https://github.com/Genivia/ugrep - Note that `.' matches any non-newline character. Pattern `\n' matches- a newline character. Multiple lines may be matched with patterns that+ Note that `.' matches any non-newline character. Pattern `\n' matches+ a newline character. Multiple lines may be matched with patterns that match one or more newline characters. EXIT STATUS@@ -4417,54 +4521,54 @@ in markdown: >1 An error occurred. - If -q or --quiet or --silent is used and a line is selected, the exit+ If -q or --quiet or --silent is used and a line is selected, the exit status is 0 even if an error occurred. CONFIGURATION- The ug command is intended for context-dependent interactive searching- and is equivalent to the ugrep --config command to load the default+ The ug command is intended for context-dependent interactive searching+ and is equivalent to the ugrep --config command to load the default configuration file `.ugrep' when present in the working directory or in the home directory. A configuration file contains `NAME=VALUE' pairs per line, where `NAME`- is the name of a long option (without `--') and `=VALUE' is an argu-- ment, which is optional and may be omitted depending on the option.+ is the name of a long option (without `--') and `=VALUE' is an argu-+ ment, which is optional and may be omitted depending on the option. Empty lines and lines starting with a `#' are ignored. - The --config=FILE option and its abbreviated form ---FILE load the- specified configuration file located in the working directory or, when- not found, located in the home directory. An error is produced when+ The --config=FILE option and its abbreviated form ---FILE load the+ specified configuration file located in the working directory or, when+ not found, located in the home directory. An error is produced when FILE is not found or cannot be read. - Command line options are parsed in the following order: the configura-- tion file is loaded first, followed by the remaining options and argu-+ Command line options are parsed in the following order: the configura-+ tion file is loaded first, followed by the remaining options and argu- ments on the command line. - The --save-config option saves a `.ugrep' configuration file to the- working directory with a subset of the current options. The --save-- config=FILE option saves the configuration to FILE. The configuration+ The --save-config option saves a `.ugrep' configuration file to the+ working directory with a subset of the current options. The --save-+ config=FILE option saves the configuration to FILE. The configuration is written to standard output when FILE is a `-'. GLOBBING- Globbing is used by options -g, --include, --include-dir, --include-- from, --exclude, --exclude-dir, --exclude-from and --ignore-files to- match pathnames and basenames in recursive searches. Glob arguments+ Globbing is used by options -g, --include, --include-dir, --include-+ from, --exclude, --exclude-dir, --exclude-from and --ignore-files to+ match pathnames and basenames in recursive searches. Glob arguments for these options should be quoted to prevent shell globbing. - Globbing supports gitignore syntax and the corresponding matching- rules, except that a glob normally matches files but not directories.+ Globbing supports gitignore syntax and the corresponding matching+ rules, except that a glob normally matches files but not directories. If a glob ends in a path separator `/', then it matches directories but- not files, as if --include-dir or --exclude-dir is specified. When a+ not files, as if --include-dir or --exclude-dir is specified. When a glob contains a path separator `/', the full pathname is matched. Oth-- erwise the basename of a file or directory is matched. For example,- *.h matches foo.h and bar/foo.h. bar/*.h matches bar/foo.h but not- foo.h and not bar/bar/foo.h. Use a leading `/' to force /*.h to match+ erwise the basename of a file or directory is matched. For example,+ *.h matches foo.h and bar/foo.h. bar/*.h matches bar/foo.h but not+ foo.h and not bar/bar/foo.h. Use a leading `/' to force /*.h to match foo.h but not bar/foo.h. - When a glob starts with a `^' or a `!' as in -g^GLOB, the match is+ When a glob starts with a `^' or a `!' as in -g^GLOB, the match is negated. Likewise, a `!' (but not a `^') may be used with globs in the- files specified --include-from, --exclude-from, and --ignore-files to- negate the glob match. Empty lines or lines starting with a `#' are+ files specified --include-from, --exclude-from, and --ignore-files to+ negate the glob match. Empty lines or lines starting with a `#' are ignored. Glob Syntax and Conventions@@ -4479,12 +4583,12 @@ in markdown: [!a-z] Matches one character not in the selected range of characters. - / When used at the begin of a glob, matches if pathname has no /.+ / When used at the begin of a glob, matches if pathname has no /. When used at the end of a glob, matches directories only. **/ Matches zero or more directories. - /** When used at the end of a glob, matches everything after the /.+ /** When used at the end of a glob, matches everything after the /. \? Matches a ? (or any character specified after the backslash). @@ -4521,46 +4625,46 @@ in markdown: a\?b Matches a?b, but not a, b, ab, axb, a/b - Note that exclude glob patterns take priority over include glob pat-- terns when specified with options -g, --exclude, --exclude-dir,+ Note that exclude glob patterns take priority over include glob pat-+ terns when specified with options -g, --exclude, --exclude-dir, --include and include-dir. - Glob patterns specified with prefix `!' in any of the files associated- with --include-from, --exclude-from and --ignore-files will negate a- previous glob match. That is, any matching file or directory excluded- by a previous glob pattern specified in the files associated with- --exclude-from or --ignore-file will become included again. Likewise,- any matching file or directory included by a previous glob pattern- specified in the files associated with --include-from will become+ Glob patterns specified with prefix `!' in any of the files associated+ with --include-from, --exclude-from and --ignore-files will negate a+ previous glob match. That is, any matching file or directory excluded+ by a previous glob pattern specified in the files associated with+ --exclude-from or --ignore-file will become included again. Likewise,+ any matching file or directory included by a previous glob pattern+ specified in the files associated with --include-from will become excluded again. ENVIRONMENT GREP_PATH- May be used to specify a file path to pattern files. The file- path is used by option -f to open a pattern file, when the pat-+ May be used to specify a file path to pattern files. The file+ path is used by option -f to open a pattern file, when the pat- tern file does not exist. GREP_COLOR- May be used to specify ANSI SGR parameters to highlight matches- when option --color is used, e.g. 1;35;40 shows pattern matches+ May be used to specify ANSI SGR parameters to highlight matches+ when option --color is used, e.g. 1;35;40 shows pattern matches in bold magenta text on a black background. Deprecated in favor of GREP_COLORS, but still supported. GREP_COLORS- May be used to specify ANSI SGR parameters to highlight matches- and other attributes when option --color is used. Its value is- a colon-separated list of ANSI SGR parameters that defaults to+ May be used to specify ANSI SGR parameters to highlight matches+ and other attributes when option --color is used. Its value is+ a colon-separated list of ANSI SGR parameters that defaults to cx=33:mt=1;31:fn=1;35:ln=1;32:cn=1;32:bn=1;32:se=36. The mt=,- ms=, and mc= capabilities of GREP_COLORS take priority over+ ms=, and mc= capabilities of GREP_COLORS take priority over GREP_COLOR. Option --colors takes priority over GREP_COLORS. GREP_COLORS- Colors are specified as string of colon-separated ANSI SGR parameters- of the form `what=substring', where `substring' is a semicolon-sepa-- rated list of ANSI SGR codes or `k' (black), `r' (red), `g' (green),- `y' (yellow), `b' (blue), `m' (magenta), `c' (cyan), `w' (white).- Upper case specifies background colors. A `+' qualifies a color as- bright. A foreground and a background color may be combined with one+ Colors are specified as string of colon-separated ANSI SGR parameters+ of the form `what=substring', where `substring' is a semicolon-sepa-+ rated list of ANSI SGR codes or `k' (black), `r' (red), `g' (green),+ `y' (yellow), `b' (blue), `m' (magenta), `c' (cyan), `w' (white).+ Upper case specifies background colors. A `+' qualifies a color as+ bright. A foreground and a background color may be combined with one or more font properties `n' (normal), `f' (faint), `h' (highlight), `i' (invert), `u' (underline). Substrings may be specified for: @@ -4572,10 +4676,10 @@ in markdown: mt= SGR substring for matching text in any matching line. - ms= SGR substring for matching text in a selected line. The sub-+ ms= SGR substring for matching text in a selected line. The sub- string mt= by default. - mc= SGR substring for matching text in a context line. The sub-+ mc= SGR substring for matching text in a context line. The sub- string mt= by default. fn= SGR substring for filenames.@@ -4590,12 +4694,12 @@ in markdown: rv a Boolean parameter, switches sl= and cx= with option -v. - hl a Boolean parameter, enables filename hyperlinks (\33]8;;link).+ hl a Boolean parameter, enables filename hyperlinks (\33]8;;link). ne a Boolean parameter, disables ``erase in line'' \33[K. FORMAT- Option --format=FORMAT specifies an output format for file matches.+ Option --format=FORMAT specifies an output format for file matches. Fields may be used in FORMAT, which expand into the following values: %[ARG]F@@ -4695,53 +4799,53 @@ in markdown: %u select unique lines only, unless option -u is used. - %1 the first regex group capture of the match, and so on up to+ %1 the first regex group capture of the match, and so on up to group %9, same as %[1]#; requires option -P. %[NUM]# the regex group capture NUM; requires option -P. %[NUM]b- the byte offset of the group capture NUM; requires option -P.+ the byte offset of the group capture NUM; requires option -P. Use e for the ending byte offset and d for the byte length. %[NUM1|NUM2|...]# the first group capture NUM that matched; requires option -P. %[NUM1|NUM2|...]b- the byte offset of the first group capture NUM that matched;- requires option -P. Use e for the ending byte offset and d for+ the byte offset of the first group capture NUM that matched;+ requires option -P. Use e for the ending byte offset and d for the byte length. %[NAME]#- the NAMEd group capture; requires option -P and capturing pat-+ the NAMEd group capture; requires option -P and capturing pat- tern `(?<NAME>PATTERN)', see also %G. %[NAME]b- the byte offset of the NAMEd group capture; requires option -P- and capturing pattern `(?<NAME>PATTERN)'. Use e for the ending+ the byte offset of the NAMEd group capture; requires option -P+ and capturing pattern `(?<NAME>PATTERN)'. Use e for the ending byte offset and d for the byte length. %[NAME1|NAME2|...]#- the first NAMEd group capture that matched; requires option -P+ the first NAMEd group capture that matched; requires option -P and capturing pattern `(?<NAME>PATTERN)', see also %G. %[NAME1|NAME2|...]b- the byte offset of the first NAMEd group capture that matched;- requires option -P and capturing pattern `(?<NAME>PATTERN)'.+ the byte offset of the first NAMEd group capture that matched;+ requires option -P and capturing pattern `(?<NAME>PATTERN)'. Use e for the ending byte offset and d for the byte length. - %G list of group capture indices/names that matched; requires+ %G list of group capture indices/names that matched; requires option -P. %[TEXT1|TEXT2|...]G- list of TEXT indexed by group capture indices that matched;+ list of TEXT indexed by group capture indices that matched; requires option -P. %g the group capture index/name matched or 1; requires option -P. %[TEXT1|TEXT2|...]g- the first TEXT indexed by the first group capture index that+ the first TEXT indexed by the first group capture index that matched; requires option -P. %% the percentage sign.@@ -4749,22 +4853,22 @@ in markdown: Formatted output is written without a terminating newline, unless %~ or `\n' is explicitly specified in the format string. - The [ARG] part of a field is optional and may be omitted. When- present, the argument must be placed in [] brackets, for example %[,]F+ The [ARG] part of a field is optional and may be omitted. When+ present, the argument must be placed in [] brackets, for example %[,]F to output a comma, the pathname, and a separator. %[SEP]$ and %u are switches and do not send anything to the output. - The separator used by the %F, %H, %N, %K, %B, %S and %G fields may be+ The separator used by the %F, %H, %N, %K, %B, %S and %G fields may be changed by preceding the field by %[SEP]$. When [SEP] is not provided,- this reverts the separator to the default separator or the separator+ this reverts the separator to the default separator or the separator specified with --separator. Formatted output is written for each matching pattern, which means that- a line may be output multiple times when patterns match more than once- on the same line. If field %u is specified anywhere in a format+ a line may be output multiple times when patterns match more than once+ on the same line. If field %u is specified anywhere in a format string, matching lines are output only once, unless option -u,- --ungroup is specified or when more than one line of input matched the+ --ungroup is specified or when more than one line of input matched the search pattern. Additional formatting options:@@ -4781,8 +4885,8 @@ in markdown: --format-end=FORMAT the FORMAT when ending the search. - The context options -A, -B, -C, -y, and display options --break,- --heading, --color, -T, and --null have no effect on formatted output.+ The context options -A, -B, -C, -y, and display options --break,+ --heading, --color, -T, and --null have no effect on formatted output. EXAMPLES Display lines containing the word `patricia' in `myfile.txt':@@ -4843,7 +4947,7 @@ in markdown: $ ugrep -n -f c++/comments myfile.cpp - List the lines that need fixing in a C/C++ source file by looking for+ List the lines that need fixing in a C/C++ source file by looking for the word `FIXME' while skipping any `FIXME' in quoted strings: $ ugrep -e FIXME -N '"(\\.|\\\r?\n|[^\\\n"])*"' myfile.cpp@@ -4873,7 +4977,7 @@ in markdown: $ ugrep -z -tc++ -n FIXME project.tgz - Recursively find lines with `FIXME' in C/C++ files, but do not search+ Recursively find lines with `FIXME' in C/C++ files, but do not search any `bak' and `old' directories: $ ugrep -n FIXME -tc++ -g^bak/,^old/@@ -4883,9 +4987,9 @@ in markdown: $ ugrep -z -w --filter='pdf:pdftotext % -' copyright - Match the binary pattern `A3hhhhA3hh' (hex) in a binary file without- Unicode pattern matching -U (which would otherwise match `\xaf' as a- Unicode character U+00A3 with UTF-8 byte sequence C2 A3) and display+ Match the binary pattern `A3hhhhA3hh' (hex) in a binary file without+ Unicode pattern matching -U (which would otherwise match `\xaf' as a+ Unicode character U+00A3 with UTF-8 byte sequence C2 A3) and display the results in hex with -X using `less -R' as a pager: $ ugrep --pager -UXo '\xa3[\x00-\xff]{2}\xa3[\x00-\xff]' a.out@@ -4898,12 +5002,12 @@ in markdown: $ ugrep -l '' --ignore-files - List all files containing a RPM signature, located in the `rpm' direc-+ List all files containing a RPM signature, located in the `rpm' direc- tory and recursively below up to two levels deeper (3 levels total): $ ugrep -3 -l -tRpm '' rpm/ - Monitor the system log for bug reports and ungroup multiple matches on+ Monitor the system log for bug reports and ungroup multiple matches on a line: $ tail -f /var/log/system.log | ugrep -u -i -w bug@@ -4927,8 +5031,8 @@ in markdown: LICENSE- ugrep is released under the BSD-3 license. All parts of the software- have reasonable copyright terms permitting free redistribution. This+ ugrep is released under the BSD-3 license. All parts of the software+ have reasonable copyright terms permitting free redistribution. This includes the ability to reuse all or parts of the ugrep source tree. SEE ALSO@@ -4936,7 +5040,7 @@ in markdown: - ugrep 3.5.0 January 05, 2022 UGREP(1)+ ugrep 3.6.0 January 11, 2022 UGREP(1) 🔝 [Back to table of contents](#toc)
@@ -1,6 +1,6 @@ #! /bin/sh # Guess values for system-dependent variables and create Makefiles.-# Generated by GNU Autoconf 2.71 for ugrep 3.4.+# Generated by GNU Autoconf 2.71 for ugrep 3.6. # # # Copyright (C) 1992-1996, 1998-2017, 2020-2021 Free Software Foundation,@@ -10,7 +10,7 @@ # This configure script is free software; the Free Software Foundation # gives unlimited permission to copy, distribute and modify it. #-# Copyright (C) 2019-2021 Robert van Engelen, Genivia Inc.+# Copyright (C) 2019-2022 Robert van Engelen, Genivia Inc. ## -------------------- ## ## M4sh Initialization. ## ## -------------------- ##@@ -609,8 +609,8 @@ MAKEFLAGS= # Identity of this package. PACKAGE_NAME='ugrep' PACKAGE_TARNAME='ugrep'-PACKAGE_VERSION='3.4'-PACKAGE_STRING='ugrep 3.4'+PACKAGE_VERSION='3.6'+PACKAGE_STRING='ugrep 3.6' PACKAGE_BUGREPORT='' PACKAGE_URL='' @@ -1349,7 +1349,7 @@ if test "$ac_init_help" = "long"; then # Omit some internal or obsolete options to make the list less imposing. # This message is too long to be a string in the A/UX 3.1 sh. cat <<_ACEOF-\`configure' configures ugrep 3.4 to adapt to many kinds of systems.+\`configure' configures ugrep 3.6 to adapt to many kinds of systems. Usage: $0 [OPTION]... [VAR=VALUE]... @@ -1420,7 +1420,7 @@ fi if test -n "$ac_init_help"; then case $ac_init_help in- short | recursive ) echo "Configuration of ugrep 3.4:";;+ short | recursive ) echo "Configuration of ugrep 3.6:";; esac cat <<\_ACEOF @@ -1562,14 +1562,14 @@ fi test -n "$ac_init_help" && exit $ac_status if $ac_init_version; then cat <<\_ACEOF-ugrep configure 3.4+ugrep configure 3.6 generated by GNU Autoconf 2.71 Copyright (C) 2021 Free Software Foundation, Inc. This configure script is free software; the Free Software Foundation gives unlimited permission to copy, distribute and modify it. -Copyright (C) 2019-2021 Robert van Engelen, Genivia Inc.+Copyright (C) 2019-2022 Robert van Engelen, Genivia Inc. _ACEOF exit fi@@ -2099,7 +2099,7 @@ cat >config.log <<_ACEOF This file contains any messages produced by compilers while running configure, to aid debugging if configure makes a mistake. -It was created by ugrep $as_me 3.4, which was+It was created by ugrep $as_me 3.6, which was generated by GNU Autoconf 2.71. Invocation command line was $ $0$ac_configure_args_raw@@ -3586,7 +3586,7 @@ fi # Define the identity of the package. PACKAGE='ugrep'- VERSION='3.4'+ VERSION='3.6' printf "%s\n" "#define PACKAGE \"$PACKAGE\"" >>confdefs.h@@ -9358,7 +9358,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1 # report actual input values of CONFIG_FILES etc. instead of their # values after options handling. ac_log="-This file was extended by ugrep $as_me 3.4, which was+This file was extended by ugrep $as_me 3.6, which was generated by GNU Autoconf 2.71. Invocation command line was CONFIG_FILES = $CONFIG_FILES@@ -9426,7 +9426,7 @@ ac_cs_config_escaped=`printf "%s\n" "$ac_cs_config" | sed "s/^ //; s/'/'\\\\\\\\ cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 ac_cs_config='$ac_cs_config_escaped' ac_cs_version="\\-ugrep config.status 3.4+ugrep config.status 3.6 configured by $0, generated by GNU Autoconf 2.71, with options \\"\$ac_cs_config\\"
@@ -1,7 +1,7 @@-AC_INIT([ugrep],[3.4])+AC_INIT([ugrep],[3.6]) AM_INIT_AUTOMAKE([foreign]) AC_CONFIG_HEADERS([config.h])-AC_COPYRIGHT([Copyright (C) 2019-2021 Robert van Engelen, Genivia Inc.])+AC_COPYRIGHT([Copyright (C) 2019-2022 Robert van Engelen, Genivia Inc.]) AC_CONFIG_MACRO_DIR([m4])
@@ -30,7 +30,7 @@ @file absmatcher.h @brief RE/flex abstract matcher base class and pattern matcher class @author Robert van Engelen - engelen@genivia.com-@copyright (c) 2016-2020, Robert van Engelen, Genivia Inc. All rights reserved.+@copyright (c) 2016-2022, Robert van Engelen, Genivia Inc. All rights reserved. @copyright (c) BSD-3 License - see LICENSE.txt */ @@ -106,6 +106,11 @@ class AbstractMatcher { static const size_t BUFSZ = (64*1024); ///< initial buffer size, at least 4096 bytes #else static const size_t BUFSZ = REFLEX_BUFSZ;+#endif+#ifndef REFLEX_BOLSZ+ static const size_t BOLSZ = (256*1024); ///< max begin of line size till match to retain in memory by growing the buffer+#else+ static const size_t BOLSZ = REFLEX_BOLSZ; #endif static const size_t BLOCK = 4096; ///< minimum remaining unused space in the buffer, to prevent excessive shifting static const size_t REDO = 0x7FFFFFFF; ///< reflex::Matcher::accept() returns "redo" with reflex::Matcher option "A"@@ -626,15 +631,15 @@ class AbstractMatcher { const char *t = txt_; size_t n = 0; #if defined(HAVE_AVX512BW) && (!defined(_MSC_VER) || defined(_WIN64))- if (s + 63 > t && have_HW_AVX512BW())+ if (have_HW_AVX512BW()) { n += simd_nlcount_avx512bw(s, t); }- else if (s + 31 > t && have_HW_AVX2())+ else if (have_HW_AVX2()) { n += simd_nlcount_avx2(s, t); }- else if (have_HW_SSE2())+ else { __m128i vlcn = _mm_set1_epi8('\n'); while (s + 15 <= t)@@ -647,11 +652,11 @@ class AbstractMatcher { } } #elif defined(HAVE_AVX2)- if (s + 31 > t && have_HW_AVX2())+ if (have_HW_AVX2()) { n += simd_nlcount_avx2(s, t); }- else if (have_HW_SSE2())+ else { __m128i vlcn = _mm_set1_epi8('\n'); while (s + 15 <= t)@@ -664,17 +669,14 @@ class AbstractMatcher { } } #elif defined(HAVE_SSE2)- if (have_HW_SSE2())+ __m128i vlcn = _mm_set1_epi8('\n');+ while (s + 15 <= t) {- __m128i vlcn = _mm_set1_epi8('\n');- while (s + 15 <= t)- {- __m128i vlcm = _mm_loadu_si128(reinterpret_cast<const __m128i*>(s));- __m128i vlceq = _mm_cmpeq_epi8(vlcm, vlcn);- uint32_t mask = _mm_movemask_epi8(vlceq);- n += popcount(mask);- s += 16;- }+ __m128i vlcm = _mm_loadu_si128(reinterpret_cast<const __m128i*>(s));+ __m128i vlceq = _mm_cmpeq_epi8(vlcm, vlcn);+ uint32_t mask = _mm_movemask_epi8(vlceq);+ n += popcount(mask);+ s += 16; } #elif defined(HAVE_NEON) {@@ -1385,7 +1387,7 @@ class AbstractMatcher { #if defined(WITH_SPAN) (void)lineno(); cno_ = 0;- if (bol_ + Const::BUFSZ - buf_ < txt_ - bol_ && evh_ == NULL)+ if (bol_ + Const::BOLSZ - buf_ < txt_ - bol_ && evh_ == NULL) { // this line is very long, so shift all the way to the match instead of to the begin of the last line DBGLOG("Line in buffer is too long to shift, moving bol position to text match position");
@@ -30,7 +30,7 @@ @file matcher.h @brief RE/flex matcher engine @author Robert van Engelen - engelen@genivia.com-@copyright (c) 2016-2020, Robert van Engelen, Genivia Inc. All rights reserved.+@copyright (c) 2016-2022, Robert van Engelen, Genivia Inc. All rights reserved. @copyright (c) BSD-3 License - see LICENSE.txt */ @@ -396,530 +396,21 @@ class Matcher : public PatternMatcher<reflex::Pattern> { /// Returns true if input matched the pattern using method Const::SCAN, Const::FIND, Const::SPLIT, or Const::MATCH. virtual size_t match(Method method) ///< Const::SCAN, Const::FIND, Const::SPLIT, or Const::MATCH /// @returns nonzero if input matched the pattern- {- DBGLOG("BEGIN Matcher::match()");- reset_text();- len_ = 0; // split text length starts with 0- anc_ = false; // no word boundary anchor found and applied-scan:- txt_ = buf_ + cur_;-#if !defined(WITH_NO_INDENT)- mrk_ = false;- ind_ = pos_; // ind scans input in buf[] in newline() up to pos - 1- col_ = 0; // count columns for indent matching-#endif-find:- int c1 = got_;- bool bol = at_bol(); // at begin of line?- if (pat_->fsm_ != NULL)- fsm_.c1 = c1;-#if !defined(WITH_NO_INDENT)-redo:-#endif- lap_.resize(0);- cap_ = 0;- bool nul = method == Const::MATCH;- if (pat_->fsm_ != NULL)- {- DBGLOG("FSM code %p", pat_->fsm_);- fsm_.bol = bol;- fsm_.nul = nul;- pat_->fsm_(*this);- nul = fsm_.nul;- c1 = fsm_.c1;- }- else if (pat_->opc_ != NULL)- {- const Pattern::Opcode *pc = pat_->opc_;- while (true)- {- Pattern::Opcode opcode = *pc;- DBGLOG("Fetch: code[%zu] = 0x%08X", pc - pat_->opc_, opcode);- if (!Pattern::is_opcode_goto(opcode))- {- switch (opcode >> 24)- {- case 0xFE: // TAKE- cap_ = Pattern::long_index_of(opcode);- cur_ = pos_;- ++pc;- DBGLOG("Take: cap = %zu", cap_);- continue;- case 0xFD: // REDO- cap_ = Const::REDO;- DBGLOG("Redo");- cur_ = pos_;- ++pc;- continue;- case 0xFC: // TAIL- {- Pattern::Lookahead la = Pattern::lookahead_of(opcode);- DBGLOG("Tail: %u", la);- if (lap_.size() > la && lap_[la] >= 0)- cur_ = txt_ - buf_ + static_cast<size_t>(lap_[la]); // mind the (new) gap- ++pc;- continue;- }- case 0xFB: // HEAD- {- Pattern::Lookahead la = Pattern::lookahead_of(opcode);- DBGLOG("Head: lookahead[%u] = %zu", la, pos_ - (txt_ - buf_));- if (lap_.size() <= la)- lap_.resize(la + 1, -1);- lap_[la] = static_cast<int>(pos_ - (txt_ - buf_)); // mind the gap- ++pc;- continue;- }-#if !defined(WITH_NO_INDENT)- case Pattern::META_DED - Pattern::META_MIN:- if (ded_ > 0)- {- Pattern::Index jump = Pattern::index_of(opcode);- if (jump == Pattern::Const::LONG)- jump = Pattern::long_index_of(pc[1]);- DBGLOG("Dedent ded = %zu", ded_); // unconditional dedent matching \j- nul = true;- pc = pat_->opc_ + jump;- continue;- }-#endif- }- if (c1 == EOF)- break;- int c0 = c1;- c1 = get();- DBGLOG("Get: c1 = %d", c1);- Pattern::Index back = Pattern::Const::IMAX; // where to jump back to (backtrack on meta transitions)- Pattern::Index jump = Pattern::Const::IMAX; // to jump to longest sequence of matching metas- while (true)- {- if ((jump == Pattern::Const::IMAX || back == Pattern::Const::IMAX) && !Pattern::is_opcode_goto(opcode))- {- // we no longer have to pass through all if jump and back are set- switch (opcode >> 24)- {- case 0xFE: // TAKE- cap_ = Pattern::long_index_of(opcode);- cur_ = pos_;- if (c1 != EOF)- --cur_; // must unget one char- opcode = *++pc;- DBGLOG("Take: cap = %zu", cap_);- continue;- case 0xFD: // REDO- cap_ = Const::REDO;- DBGLOG("Redo");- cur_ = pos_;- if (c1 != EOF)- --cur_; // must unget one char- opcode = *++pc;- continue;- case 0xFC: // TAIL- {- Pattern::Lookahead la = Pattern::lookahead_of(opcode);- DBGLOG("Tail: %u", la);- if (lap_.size() > la && lap_[la] >= 0)- cur_ = txt_ - buf_ + static_cast<size_t>(lap_[la]); // mind the (new) gap- opcode = *++pc;- continue;- }- case 0xFB: // HEAD- opcode = *++pc;- continue;-#if !defined(WITH_NO_INDENT)- case Pattern::META_DED - Pattern::META_MIN:- DBGLOG("DED? %d", c1);- if (jump == Pattern::Const::IMAX && back == Pattern::Const::IMAX && bol && dedent())- {- jump = Pattern::index_of(opcode);- if (jump == Pattern::Const::LONG)- jump = Pattern::long_index_of(*++pc);- }- opcode = *++pc;- continue;- case Pattern::META_IND - Pattern::META_MIN:- DBGLOG("IND? %d", c1);- if (jump == Pattern::Const::IMAX && back == Pattern::Const::IMAX && bol && indent())- {- jump = Pattern::index_of(opcode);- if (jump == Pattern::Const::LONG)- jump = Pattern::long_index_of(*++pc);- }- opcode = *++pc;- continue;- case Pattern::META_UND - Pattern::META_MIN:- DBGLOG("UND");- if (mrk_)- {- jump = Pattern::index_of(opcode);- if (jump == Pattern::Const::LONG)- jump = Pattern::long_index_of(*++pc);- }- mrk_ = false;- ded_ = 0;- opcode = *++pc;- continue;-#endif- case Pattern::META_EOB - Pattern::META_MIN:- DBGLOG("EOB? %d", c1);- if (jump == Pattern::Const::IMAX && c1 == EOF)- {- jump = Pattern::index_of(opcode);- if (jump == Pattern::Const::LONG)- jump = Pattern::long_index_of(*++pc);- }- opcode = *++pc;- continue;- case Pattern::META_BOB - Pattern::META_MIN:- DBGLOG("BOB? %d", at_bob());- if (jump == Pattern::Const::IMAX && at_bob())- {- jump = Pattern::index_of(opcode);- if (jump == Pattern::Const::LONG)- jump = Pattern::long_index_of(*++pc);- }- opcode = *++pc;- continue;- case Pattern::META_EOL - Pattern::META_MIN:- DBGLOG("EOL? %d", c1);- anc_ = true;- if (jump == Pattern::Const::IMAX && (c1 == EOF || c1 == '\n' || (c1 == '\r' && peek() == '\n')))- {- jump = Pattern::index_of(opcode);- if (jump == Pattern::Const::LONG)- jump = Pattern::long_index_of(*++pc);- }- opcode = *++pc;- continue;- case Pattern::META_BOL - Pattern::META_MIN:- DBGLOG("BOL? %d", bol);- anc_ = true;- if (jump == Pattern::Const::IMAX && bol)- {- jump = Pattern::index_of(opcode);- if (jump == Pattern::Const::LONG)- jump = Pattern::long_index_of(*++pc);- }- opcode = *++pc;- continue;- case Pattern::META_EWE - Pattern::META_MIN:- DBGLOG("EWE? %d %d %d", c0, c1, isword(c0) && !isword(c1));- anc_ = true;- if (jump == Pattern::Const::IMAX && (isword(c0) || opt_.W) && !isword(c1))- {- jump = Pattern::index_of(opcode);- if (jump == Pattern::Const::LONG)- jump = Pattern::long_index_of(*++pc);- }- opcode = *++pc;- continue;- case Pattern::META_BWE - Pattern::META_MIN:- DBGLOG("BWE? %d %d %d", c0, c1, !isword(c0) && isword(c1));- anc_ = true;- if (jump == Pattern::Const::IMAX && !isword(c0) && isword(c1))- {- jump = Pattern::index_of(opcode);- if (jump == Pattern::Const::LONG)- jump = Pattern::long_index_of(*++pc);- }- opcode = *++pc;- continue;- case Pattern::META_EWB - Pattern::META_MIN:- DBGLOG("EWB? %d", at_eow());- anc_ = true;- if (jump == Pattern::Const::IMAX && isword(got_) &&- !isword(static_cast<unsigned char>(txt_[len_])))- {- jump = Pattern::index_of(opcode);- if (jump == Pattern::Const::LONG)- jump = Pattern::long_index_of(*++pc);- }- opcode = *++pc;- continue;- case Pattern::META_BWB - Pattern::META_MIN:- DBGLOG("BWB? %d", at_bow());- anc_ = true;- if (jump == Pattern::Const::IMAX && !isword(got_) &&- (opt_.W || isword(static_cast<unsigned char>(txt_[len_]))))- {- jump = Pattern::index_of(opcode);- if (jump == Pattern::Const::LONG)- jump = Pattern::long_index_of(*++pc);- }- opcode = *++pc;- continue;- case Pattern::META_NWE - Pattern::META_MIN:- DBGLOG("NWE? %d %d %d", c0, c1, isword(c0) == isword(c1));- anc_ = true;- if (jump == Pattern::Const::IMAX && isword(c0) == isword(c1))- {- jump = Pattern::index_of(opcode);- if (jump == Pattern::Const::LONG)- jump = Pattern::long_index_of(*++pc);- }- opcode = *++pc;- continue;- case Pattern::META_NWB - Pattern::META_MIN:- DBGLOG("NWB? %d %d", at_bow(), at_eow());- anc_ = true;- if (jump == Pattern::Const::IMAX &&- isword(got_) == isword(static_cast<unsigned char>(txt_[len_])))- {- jump = Pattern::index_of(opcode);- if (jump == Pattern::Const::LONG)- jump = Pattern::long_index_of(*++pc);- }- opcode = *++pc;- continue;- case 0xFF: // LONG- opcode = *++pc;- continue;- }- }- if (jump == Pattern::Const::IMAX)- {- if (back != Pattern::Const::IMAX)- {- pc = pat_->opc_ + back;- opcode = *pc;- }- break;- }- DBGLOG("Backtrack: pc = %u", jump);- if (back == Pattern::Const::IMAX)- back = static_cast<Pattern::Index>(pc - pat_->opc_);- pc = pat_->opc_ + jump;- opcode = *pc;- jump = Pattern::Const::IMAX;- }- if (c1 == EOF)- break;- }- else- {- if (Pattern::is_opcode_halt(opcode))- break;- if (c1 == EOF)- break;- c1 = get();- DBGLOG("Get: c1 = %d", c1);- if (c1 == EOF)- break;- }- Pattern::Opcode lo = c1 << 24;- Pattern::Opcode hi = lo | 0x00FFFFFF;-unrolled:- if (hi < opcode || lo > (opcode << 8))- {- opcode = *++pc;- if (hi < opcode || lo > (opcode << 8))- {- opcode = *++pc;- if (hi < opcode || lo > (opcode << 8))- {- opcode = *++pc;- if (hi < opcode || lo > (opcode << 8))- {- opcode = *++pc;- if (hi < opcode || lo > (opcode << 8))- {- opcode = *++pc;- if (hi < opcode || lo > (opcode << 8))- {- opcode = *++pc;- if (hi < opcode || lo > (opcode << 8))- {- opcode = *++pc;- if (hi < opcode || lo > (opcode << 8))- {- opcode = *++pc;- goto unrolled;- }- }- }- }- }- }- }- }- Pattern::Index jump = Pattern::index_of(opcode);- if (jump == 0)- {- // loop back to start state after only one char matched (one transition) but w/o full match, then optimize- if (cap_ == 0 && pos_ == cur_ + 1 && method == Const::FIND)- cur_ = pos_; // set cur_ to move forward from cur_ + 1 with FIND advance()- }- else if (jump >= Pattern::Const::LONG)- {- if (jump == Pattern::Const::HALT)- break;- jump = Pattern::long_index_of(pc[1]);- }- pc = pat_->opc_ + jump;- }- }-#if !defined(WITH_NO_INDENT)- if (mrk_ && cap_ != Const::REDO)- {- if (col_ > 0 && (tab_.empty() || tab_.back() < col_))- {- DBGLOG("Set new stop: tab_[%zu] = %zu", tab_.size(), col_);- tab_.push_back(col_);- }- else if (!tab_.empty() && tab_.back() > col_)- {- size_t n;- for (n = tab_.size() - 1; n > 0; --n)- if (tab_.at(n - 1) <= col_)- break;- ded_ += tab_.size() - n;- DBGLOG("Dedents: ded = %zu tab_ = %zu", ded_, tab_.size());- tab_.resize(n);- // adjust stop when indents are not aligned (Python would give an error though)- if (n > 0)- tab_.back() = col_;- }- }- if (ded_ > 0)- {- DBGLOG("Dedents: ded = %zu", ded_);- if (col_ == 0 && bol)- {- ded_ += tab_.size();- tab_.resize(0);- DBGLOG("Rescan for pending dedents: ded = %zu", ded_);- pos_ = ind_;- // avoid looping, match \j exactly- bol = false;- goto redo;- }- --ded_;- }-#endif- if (method == Const::SPLIT)- {- DBGLOG("Split: len = %zu cap = %zu cur = %zu pos = %zu end = %zu txt-buf = %zu eob = %d got = %d", len_, cap_, cur_, pos_, end_, txt_-buf_, (int)eof_, got_);- if (cap_ == 0 || (cur_ == static_cast<size_t>(txt_ - buf_) && !at_bob()))- {- if (!hit_end() && (txt_ + len_ < buf_ + end_ || peek() != EOF))- {- ++len_;- DBGLOG("Split continue: len = %zu", len_);- set_current(++cur_);- goto find;- }- if (got_ != Const::EOB)- cap_ = Const::EMPTY;- else- cap_ = 0;- set_current(end_);- got_ = Const::EOB;- DBGLOG("Split at eof: cap = %zu txt = '%s' len = %zu", cap_, std::string(txt_, len_).c_str(), len_);- DBGLOG("END Matcher::match()");- return cap_;- }- if (cur_ == 0 && at_bob() && at_end())- {- cap_ = Const::EMPTY;- got_ = Const::EOB;- }- else- {- set_current(cur_);- }- DBGLOG("Split: txt = '%s' len = %zu", std::string(txt_, len_).c_str(), len_);- DBGLOG("END Matcher::match()");- return cap_;- }- if (cap_ == 0)- {- if (method == Const::FIND)- {- if (!at_end())- {- if (anc_)- {- cur_ = txt_ - buf_; // reset current to pattern start when a word boundary was encountered- anc_ = false;- }- if (pos_ > cur_) // if we didn't fail on META alone- {- if (advance())- {- if (!pat_->one_)- goto scan;- txt_ = buf_ + cur_;- len_ = pat_->len_;- set_current(cur_ + len_);- return cap_ = 1;- }- }- txt_ = buf_ + cur_;- }- }- else- {- // SCAN and MATCH: no match: backup to begin of unmatched text to report as error- cur_ = txt_ - buf_;- }- }- len_ = cur_ - (txt_ - buf_);- if (len_ == 0 && !nul)- {- DBGLOG("Empty or no match cur = %zu pos = %zu end = %zu", cur_, pos_, end_);- pos_ = cur_;- if (at_end())- {- set_current(cur_);- DBGLOG("Reject empty match at EOF");- cap_ = 0;- }- else if (method == Const::FIND)- {- DBGLOG("Reject empty match and continue?");- // skip one char to keep searching- set_current(++cur_);- // allow FIND with "N" to match an empty line, with ^$ etc.- if (cap_ == 0 || !opt_.N)- goto scan;- DBGLOG("Accept empty match");- }- else- {- set_current(cur_);- DBGLOG("Reject empty match");- cap_ = 0;- }- }- else if (len_ == 0 && cur_ == end_)- {- DBGLOG("Hit end: got = %d", got_);- if (cap_ == Const::REDO && !opt_.A)- cap_ = 0;- }- else- {- set_current(cur_);- if (len_ > 0 && cap_ == Const::REDO && !opt_.A)- {- DBGLOG("Ignore accept and continue: len = %zu", len_);- len_ = 0;- if (method != Const::MATCH)- goto scan;- cap_ = 0;- }- }- DBGLOG("Return: cap = %zu txt = '%s' len = %zu pos = %zu got = %d", cap_, std::string(txt_, len_).c_str(), len_, pos_, got_);- DBGLOG("END match()");- return cap_;- }+ ;+ // match() with optimized AVX512BW string search scheme defined in matcher_avx512bw.cpp+ size_t simd_match_avx512bw(Method method);+ // match() with optimized AVX2 string search scheme defined in matcher_avx2.cpp+ size_t simd_match_avx2(Method method); /// Returns true if able to advance to next possible match bool advance() /// @returns true if possible match found ;- // string search scheme based on in http://0x80.pl/articles/simd-friendly-karp-rabin.html- bool simd_advance_avx2(const char*& b, const char *e, size_t &loc, size_t min, const char *pre, size_t len);- bool simd_advance_avx512bw(const char*& b, const char *e, size_t &loc, size_t min, const char *pre, size_t len);+ /// optimized AVX512BW version of advance() defined in matcher_avx512bw.cpp+ bool simd_advance_avx512bw();+ /// optimized AVX2 version of advance() defined in matcher_avx2.cpp+ bool simd_advance_avx2();+ // bool simd_advance_avx2(const char*& b, const char *e, size_t &loc, size_t min, const char *pre, size_t len);+ // bool simd_advance_avx512bw(const char*& b, const char *e, size_t &loc, size_t min, const char *pre, size_t len); #if !defined(WITH_NO_INDENT) /// Update indentation column counter for indent() and dedent(). inline void newline()@@ -951,11 +442,6 @@ unrolled: return (col_ <= 0 || (!tab_.empty() && tab_.back() >= col_)) && (tab_.empty() || tab_.back() <= col_); } #endif- /// Boyer-Moore preprocessing of the given pattern pat of length len, generates bmd_ > 0 and bms_[] shifts.- void boyer_moore_init(- const char *pat, ///< pattern string- size_t len) ///< nonzero length of the pattern string, should be less than 256- ; size_t ded_; ///< dedent count size_t col_; ///< column counter for indent matching, updated by newline(), indent(), and dedent() Stops tab_; ///< tab stops set by detecting indent margins
@@ -30,7 +30,7 @@ @file simd.h @brief RE/flex SIMD intrinsics @author Robert van Engelen - engelen@genivia.com-@copyright (c) 2016-2020, Robert van Engelen, Genivia Inc. All rights reserved.+@copyright (c) 2016-2022, Robert van Engelen, Genivia Inc. All rights reserved. @copyright (c) BSD-3 License - see LICENSE.txt */
@@ -1,6 +1,11 @@ noinst_LIBRARIES = libreflex.a libreflex_a_CPPFLAGS = -I$(top_srcdir)/include $(SIMD_FLAGS)-libreflex_a_SOURCES = $(top_srcdir)/include/reflex/abslexer.h $(top_srcdir)/include/reflex/absmatcher.h $(top_srcdir)/include/reflex/bits.h $(top_srcdir)/include/reflex/boostmatcher.h $(top_srcdir)/include/reflex/convert.h $(top_srcdir)/include/reflex/debug.h $(top_srcdir)/include/reflex/error.h $(top_srcdir)/include/reflex/flexlexer.h $(top_srcdir)/include/reflex/input.h $(top_srcdir)/include/reflex/matcher.h $(top_srcdir)/include/reflex/pattern.h $(top_srcdir)/include/reflex/posix.h $(top_srcdir)/include/reflex/ranges.h $(top_srcdir)/include/reflex/setop.h $(top_srcdir)/include/reflex/simd.h $(top_srcdir)/include/reflex/stdmatcher.h $(top_srcdir)/include/reflex/timer.h $(top_srcdir)/include/reflex/traits.h $(top_srcdir)/include/reflex/unicode.h $(top_srcdir)/include/reflex/utf8.h convert.cpp debug.cpp error.cpp input.cpp matcher.cpp pattern.cpp posix.cpp simd_avx2.cpp simd_avx512bw.cpp unicode.cpp utf8.cpp block_scripts.cpp language_scripts.cpp letter_scripts.cpp+libreflex_a_SOURCES = $(top_srcdir)/include/reflex/abslexer.h $(top_srcdir)/include/reflex/absmatcher.h $(top_srcdir)/include/reflex/bits.h $(top_srcdir)/include/reflex/boostmatcher.h $(top_srcdir)/include/reflex/convert.h $(top_srcdir)/include/reflex/debug.h $(top_srcdir)/include/reflex/error.h $(top_srcdir)/include/reflex/flexlexer.h $(top_srcdir)/include/reflex/input.h $(top_srcdir)/include/reflex/matcher.h $(top_srcdir)/include/reflex/pattern.h $(top_srcdir)/include/reflex/posix.h $(top_srcdir)/include/reflex/ranges.h $(top_srcdir)/include/reflex/setop.h $(top_srcdir)/include/reflex/simd.h $(top_srcdir)/include/reflex/stdmatcher.h $(top_srcdir)/include/reflex/timer.h $(top_srcdir)/include/reflex/traits.h $(top_srcdir)/include/reflex/unicode.h $(top_srcdir)/include/reflex/utf8.h convert.cpp debug.cpp error.cpp input.cpp matcher.cpp matcher_avx2.cpp matcher_avx512bw.cpp pattern.cpp posix.cpp simd_avx2.cpp simd_avx512bw.cpp unicode.cpp utf8.cpp block_scripts.cpp language_scripts.cpp letter_scripts.cpp -libreflex_a-simd_avx2.$(OBJEXT) : CXXFLAGS += $(SIMD_AVX2_FLAGS)-libreflex_a-simd_avx512bw.$(OBJEXT) : CXXFLAGS += $(SIMD_AVX512BW_FLAGS)+# separately compile matcher_avx2.cpp and matcher_avx512bw (with the same content as matcher.cpp) with AVX optimizations enabled+libreflex_a-matcher_avx2.$(OBJEXT) : CXXFLAGS += -DCOMPILE_AVX2 $(SIMD_AVX2_FLAGS)+libreflex_a-matcher_avx512bw.$(OBJEXT) : CXXFLAGS += -DCOMPILE_AVX512BW $(SIMD_AVX512BW_FLAGS)++# separately compile simd_avx2.cpp and simd_avx512bw.cpp with AVX optimizations enabled+libreflex_a-simd_avx2.$(OBJEXT) : CXXFLAGS += -DCOMPILE_AVX2 $(SIMD_AVX2_FLAGS)+libreflex_a-simd_avx512bw.$(OBJEXT) : CXXFLAGS += -DCOMPILE_AVX512BW $(SIMD_AVX512BW_FLAGS)
@@ -118,6 +118,8 @@ libreflex_a_LIBADD = am_libreflex_a_OBJECTS = libreflex_a-convert.$(OBJEXT) \ libreflex_a-debug.$(OBJEXT) libreflex_a-error.$(OBJEXT) \ libreflex_a-input.$(OBJEXT) libreflex_a-matcher.$(OBJEXT) \+ libreflex_a-matcher_avx2.$(OBJEXT) \+ libreflex_a-matcher_avx512bw.$(OBJEXT) \ libreflex_a-pattern.$(OBJEXT) libreflex_a-posix.$(OBJEXT) \ libreflex_a-simd_avx2.$(OBJEXT) \ libreflex_a-simd_avx512bw.$(OBJEXT) \@@ -149,6 +151,8 @@ am__depfiles_remade = ./$(DEPDIR)/libreflex_a-block_scripts.Po \ ./$(DEPDIR)/libreflex_a-language_scripts.Po \ ./$(DEPDIR)/libreflex_a-letter_scripts.Po \ ./$(DEPDIR)/libreflex_a-matcher.Po \+ ./$(DEPDIR)/libreflex_a-matcher_avx2.Po \+ ./$(DEPDIR)/libreflex_a-matcher_avx512bw.Po \ ./$(DEPDIR)/libreflex_a-pattern.Po \ ./$(DEPDIR)/libreflex_a-posix.Po \ ./$(DEPDIR)/libreflex_a-simd_avx2.Po \@@ -327,7 +331,7 @@ top_builddir = @top_builddir@ top_srcdir = @top_srcdir@ noinst_LIBRARIES = libreflex.a libreflex_a_CPPFLAGS = -I$(top_srcdir)/include $(SIMD_FLAGS)-libreflex_a_SOURCES = $(top_srcdir)/include/reflex/abslexer.h $(top_srcdir)/include/reflex/absmatcher.h $(top_srcdir)/include/reflex/bits.h $(top_srcdir)/include/reflex/boostmatcher.h $(top_srcdir)/include/reflex/convert.h $(top_srcdir)/include/reflex/debug.h $(top_srcdir)/include/reflex/error.h $(top_srcdir)/include/reflex/flexlexer.h $(top_srcdir)/include/reflex/input.h $(top_srcdir)/include/reflex/matcher.h $(top_srcdir)/include/reflex/pattern.h $(top_srcdir)/include/reflex/posix.h $(top_srcdir)/include/reflex/ranges.h $(top_srcdir)/include/reflex/setop.h $(top_srcdir)/include/reflex/simd.h $(top_srcdir)/include/reflex/stdmatcher.h $(top_srcdir)/include/reflex/timer.h $(top_srcdir)/include/reflex/traits.h $(top_srcdir)/include/reflex/unicode.h $(top_srcdir)/include/reflex/utf8.h convert.cpp debug.cpp error.cpp input.cpp matcher.cpp pattern.cpp posix.cpp simd_avx2.cpp simd_avx512bw.cpp unicode.cpp utf8.cpp block_scripts.cpp language_scripts.cpp letter_scripts.cpp+libreflex_a_SOURCES = $(top_srcdir)/include/reflex/abslexer.h $(top_srcdir)/include/reflex/absmatcher.h $(top_srcdir)/include/reflex/bits.h $(top_srcdir)/include/reflex/boostmatcher.h $(top_srcdir)/include/reflex/convert.h $(top_srcdir)/include/reflex/debug.h $(top_srcdir)/include/reflex/error.h $(top_srcdir)/include/reflex/flexlexer.h $(top_srcdir)/include/reflex/input.h $(top_srcdir)/include/reflex/matcher.h $(top_srcdir)/include/reflex/pattern.h $(top_srcdir)/include/reflex/posix.h $(top_srcdir)/include/reflex/ranges.h $(top_srcdir)/include/reflex/setop.h $(top_srcdir)/include/reflex/simd.h $(top_srcdir)/include/reflex/stdmatcher.h $(top_srcdir)/include/reflex/timer.h $(top_srcdir)/include/reflex/traits.h $(top_srcdir)/include/reflex/unicode.h $(top_srcdir)/include/reflex/utf8.h convert.cpp debug.cpp error.cpp input.cpp matcher.cpp matcher_avx2.cpp matcher_avx512bw.cpp pattern.cpp posix.cpp simd_avx2.cpp simd_avx512bw.cpp unicode.cpp utf8.cpp block_scripts.cpp language_scripts.cpp letter_scripts.cpp all: all-am .SUFFIXES:@@ -384,6 +388,8 @@ distclean-compile: @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/libreflex_a-language_scripts.Po@am__quote@ # am--include-marker @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/libreflex_a-letter_scripts.Po@am__quote@ # am--include-marker @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/libreflex_a-matcher.Po@am__quote@ # am--include-marker+@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/libreflex_a-matcher_avx2.Po@am__quote@ # am--include-marker+@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/libreflex_a-matcher_avx512bw.Po@am__quote@ # am--include-marker @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/libreflex_a-pattern.Po@am__quote@ # am--include-marker @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/libreflex_a-posix.Po@am__quote@ # am--include-marker @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/libreflex_a-simd_avx2.Po@am__quote@ # am--include-marker@@ -481,6 +487,34 @@ libreflex_a-matcher.obj: matcher.cpp @AMDEP_TRUE@@am__fastdepCXX_FALSE@ DEPDIR=$(DEPDIR) $(CXXDEPMODE) $(depcomp) @AMDEPBACKSLASH@ @am__fastdepCXX_FALSE@ $(AM_V_CXX@am__nodep@)$(CXX) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(libreflex_a_CPPFLAGS) $(CPPFLAGS) $(AM_CXXFLAGS) $(CXXFLAGS) -c -o libreflex_a-matcher.obj `if test -f 'matcher.cpp'; then $(CYGPATH_W) 'matcher.cpp'; else $(CYGPATH_W) '$(srcdir)/matcher.cpp'; fi` +libreflex_a-matcher_avx2.o: matcher_avx2.cpp+@am__fastdepCXX_TRUE@ $(AM_V_CXX)$(CXX) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(libreflex_a_CPPFLAGS) $(CPPFLAGS) $(AM_CXXFLAGS) $(CXXFLAGS) -MT libreflex_a-matcher_avx2.o -MD -MP -MF $(DEPDIR)/libreflex_a-matcher_avx2.Tpo -c -o libreflex_a-matcher_avx2.o `test -f 'matcher_avx2.cpp' || echo '$(srcdir)/'`matcher_avx2.cpp+@am__fastdepCXX_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/libreflex_a-matcher_avx2.Tpo $(DEPDIR)/libreflex_a-matcher_avx2.Po+@AMDEP_TRUE@@am__fastdepCXX_FALSE@ $(AM_V_CXX)source='matcher_avx2.cpp' object='libreflex_a-matcher_avx2.o' libtool=no @AMDEPBACKSLASH@+@AMDEP_TRUE@@am__fastdepCXX_FALSE@ DEPDIR=$(DEPDIR) $(CXXDEPMODE) $(depcomp) @AMDEPBACKSLASH@+@am__fastdepCXX_FALSE@ $(AM_V_CXX@am__nodep@)$(CXX) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(libreflex_a_CPPFLAGS) $(CPPFLAGS) $(AM_CXXFLAGS) $(CXXFLAGS) -c -o libreflex_a-matcher_avx2.o `test -f 'matcher_avx2.cpp' || echo '$(srcdir)/'`matcher_avx2.cpp++libreflex_a-matcher_avx2.obj: matcher_avx2.cpp+@am__fastdepCXX_TRUE@ $(AM_V_CXX)$(CXX) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(libreflex_a_CPPFLAGS) $(CPPFLAGS) $(AM_CXXFLAGS) $(CXXFLAGS) -MT libreflex_a-matcher_avx2.obj -MD -MP -MF $(DEPDIR)/libreflex_a-matcher_avx2.Tpo -c -o libreflex_a-matcher_avx2.obj `if test -f 'matcher_avx2.cpp'; then $(CYGPATH_W) 'matcher_avx2.cpp'; else $(CYGPATH_W) '$(srcdir)/matcher_avx2.cpp'; fi`+@am__fastdepCXX_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/libreflex_a-matcher_avx2.Tpo $(DEPDIR)/libreflex_a-matcher_avx2.Po+@AMDEP_TRUE@@am__fastdepCXX_FALSE@ $(AM_V_CXX)source='matcher_avx2.cpp' object='libreflex_a-matcher_avx2.obj' libtool=no @AMDEPBACKSLASH@+@AMDEP_TRUE@@am__fastdepCXX_FALSE@ DEPDIR=$(DEPDIR) $(CXXDEPMODE) $(depcomp) @AMDEPBACKSLASH@+@am__fastdepCXX_FALSE@ $(AM_V_CXX@am__nodep@)$(CXX) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(libreflex_a_CPPFLAGS) $(CPPFLAGS) $(AM_CXXFLAGS) $(CXXFLAGS) -c -o libreflex_a-matcher_avx2.obj `if test -f 'matcher_avx2.cpp'; then $(CYGPATH_W) 'matcher_avx2.cpp'; else $(CYGPATH_W) '$(srcdir)/matcher_avx2.cpp'; fi`++libreflex_a-matcher_avx512bw.o: matcher_avx512bw.cpp+@am__fastdepCXX_TRUE@ $(AM_V_CXX)$(CXX) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(libreflex_a_CPPFLAGS) $(CPPFLAGS) $(AM_CXXFLAGS) $(CXXFLAGS) -MT libreflex_a-matcher_avx512bw.o -MD -MP -MF $(DEPDIR)/libreflex_a-matcher_avx512bw.Tpo -c -o libreflex_a-matcher_avx512bw.o `test -f 'matcher_avx512bw.cpp' || echo '$(srcdir)/'`matcher_avx512bw.cpp+@am__fastdepCXX_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/libreflex_a-matcher_avx512bw.Tpo $(DEPDIR)/libreflex_a-matcher_avx512bw.Po+@AMDEP_TRUE@@am__fastdepCXX_FALSE@ $(AM_V_CXX)source='matcher_avx512bw.cpp' object='libreflex_a-matcher_avx512bw.o' libtool=no @AMDEPBACKSLASH@+@AMDEP_TRUE@@am__fastdepCXX_FALSE@ DEPDIR=$(DEPDIR) $(CXXDEPMODE) $(depcomp) @AMDEPBACKSLASH@+@am__fastdepCXX_FALSE@ $(AM_V_CXX@am__nodep@)$(CXX) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(libreflex_a_CPPFLAGS) $(CPPFLAGS) $(AM_CXXFLAGS) $(CXXFLAGS) -c -o libreflex_a-matcher_avx512bw.o `test -f 'matcher_avx512bw.cpp' || echo '$(srcdir)/'`matcher_avx512bw.cpp++libreflex_a-matcher_avx512bw.obj: matcher_avx512bw.cpp+@am__fastdepCXX_TRUE@ $(AM_V_CXX)$(CXX) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(libreflex_a_CPPFLAGS) $(CPPFLAGS) $(AM_CXXFLAGS) $(CXXFLAGS) -MT libreflex_a-matcher_avx512bw.obj -MD -MP -MF $(DEPDIR)/libreflex_a-matcher_avx512bw.Tpo -c -o libreflex_a-matcher_avx512bw.obj `if test -f 'matcher_avx512bw.cpp'; then $(CYGPATH_W) 'matcher_avx512bw.cpp'; else $(CYGPATH_W) '$(srcdir)/matcher_avx512bw.cpp'; fi`+@am__fastdepCXX_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/libreflex_a-matcher_avx512bw.Tpo $(DEPDIR)/libreflex_a-matcher_avx512bw.Po+@AMDEP_TRUE@@am__fastdepCXX_FALSE@ $(AM_V_CXX)source='matcher_avx512bw.cpp' object='libreflex_a-matcher_avx512bw.obj' libtool=no @AMDEPBACKSLASH@+@AMDEP_TRUE@@am__fastdepCXX_FALSE@ DEPDIR=$(DEPDIR) $(CXXDEPMODE) $(depcomp) @AMDEPBACKSLASH@+@am__fastdepCXX_FALSE@ $(AM_V_CXX@am__nodep@)$(CXX) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(libreflex_a_CPPFLAGS) $(CPPFLAGS) $(AM_CXXFLAGS) $(CXXFLAGS) -c -o libreflex_a-matcher_avx512bw.obj `if test -f 'matcher_avx512bw.cpp'; then $(CYGPATH_W) 'matcher_avx512bw.cpp'; else $(CYGPATH_W) '$(srcdir)/matcher_avx512bw.cpp'; fi`+ libreflex_a-pattern.o: pattern.cpp @am__fastdepCXX_TRUE@ $(AM_V_CXX)$(CXX) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(libreflex_a_CPPFLAGS) $(CPPFLAGS) $(AM_CXXFLAGS) $(CXXFLAGS) -MT libreflex_a-pattern.o -MD -MP -MF $(DEPDIR)/libreflex_a-pattern.Tpo -c -o libreflex_a-pattern.o `test -f 'pattern.cpp' || echo '$(srcdir)/'`pattern.cpp @am__fastdepCXX_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/libreflex_a-pattern.Tpo $(DEPDIR)/libreflex_a-pattern.Po@@ -738,6 +772,8 @@ distclean: distclean-am -rm -f ./$(DEPDIR)/libreflex_a-language_scripts.Po -rm -f ./$(DEPDIR)/libreflex_a-letter_scripts.Po -rm -f ./$(DEPDIR)/libreflex_a-matcher.Po+ -rm -f ./$(DEPDIR)/libreflex_a-matcher_avx2.Po+ -rm -f ./$(DEPDIR)/libreflex_a-matcher_avx512bw.Po -rm -f ./$(DEPDIR)/libreflex_a-pattern.Po -rm -f ./$(DEPDIR)/libreflex_a-posix.Po -rm -f ./$(DEPDIR)/libreflex_a-simd_avx2.Po@@ -797,6 +833,8 @@ maintainer-clean: maintainer-clean-am -rm -f ./$(DEPDIR)/libreflex_a-language_scripts.Po -rm -f ./$(DEPDIR)/libreflex_a-letter_scripts.Po -rm -f ./$(DEPDIR)/libreflex_a-matcher.Po+ -rm -f ./$(DEPDIR)/libreflex_a-matcher_avx2.Po+ -rm -f ./$(DEPDIR)/libreflex_a-matcher_avx512bw.Po -rm -f ./$(DEPDIR)/libreflex_a-pattern.Po -rm -f ./$(DEPDIR)/libreflex_a-posix.Po -rm -f ./$(DEPDIR)/libreflex_a-simd_avx2.Po@@ -838,8 +876,13 @@ uninstall-am: .PRECIOUS: Makefile -libreflex_a-simd_avx2.$(OBJEXT) : CXXFLAGS += $(SIMD_AVX2_FLAGS)-libreflex_a-simd_avx512bw.$(OBJEXT) : CXXFLAGS += $(SIMD_AVX512BW_FLAGS)+# separately compile matcher_avx2.cpp and matcher_avx512bw (with the same content as matcher.cpp) with AVX optimizations enabled+libreflex_a-matcher_avx2.$(OBJEXT) : CXXFLAGS += -DCOMPILE_AVX2 $(SIMD_AVX2_FLAGS)+libreflex_a-matcher_avx512bw.$(OBJEXT) : CXXFLAGS += -DCOMPILE_AVX512BW $(SIMD_AVX512BW_FLAGS)++# separately compile simd_avx2.cpp and simd_avx512bw.cpp with AVX optimizations enabled+libreflex_a-simd_avx2.$(OBJEXT) : CXXFLAGS += -DCOMPILE_AVX2 $(SIMD_AVX2_FLAGS)+libreflex_a-simd_avx512bw.$(OBJEXT) : CXXFLAGS += -DCOMPILE_AVX512BW $(SIMD_AVX512BW_FLAGS) # Tell versions [3.59,3.63) of GNU make to not export all variables. # Otherwise a system limit (for SysV at least) may be exceeded.
@@ -27,81 +27,605 @@ \******************************************************************************/ /**-@file matcher.cpp+@file matcher.cpp, matcher_avx2.cpp, matcher_avx512bw.cpp (same file) @brief RE/flex matcher engine @author Robert van Engelen - engelen@genivia.com-@copyright (c) 2016-2020, Robert van Engelen, Genivia Inc. All rights reserved.+@copyright (c) 2016-2022, Robert van Engelen, Genivia Inc. All rights reserved. @copyright (c) BSD-3 License - see LICENSE.txt */ +#if defined(COMPILE_AVX512BW) && !defined(HAVE_AVX512BW)++// appease ranlib "has no symbols"+void matcher_not_compiled_with_avx512bw() { }++#elif defined(COMPILE_AVX2) && !defined(HAVE_AVX2)++// appease ranlib "has no symbols"+void matcher_not_compiled_with_avx2() { }++#else+ #include <reflex/matcher.h> namespace reflex { -/// Boyer-Moore preprocessing of the given pattern prefix pat of length len (<=255), generates bmd_ > 0 and bms_[] shifts.-void Matcher::boyer_moore_init(const char *pat, size_t len)+/*+ The simd_match_avx512bw() and simd_match_avx2() methods are AVX-optimized+ versions of the match() method. To compile these methods separately with+ the appropriate compilation flags, this file is copied to+ matcher_avx512bw.cpp and matcher_avx2.cpp then compiled with -mavx512bw+ -DCOMPILE_AVX512BW -DHAVE_AVX512BW and with -mavx2 -DCOMPILE_AVX2+ -DHAVE_AVX2, respectively. Likewise, the simd_advance_avx512bw() and+ simd_advance_avx2() methods are optimized versions and separately compiled.+ This approach is preferred over maintaining three separate copies of source+ code files with these methods that only slightly differ. On the other hand,+ combining these versions into one source file means more #if branches.++ If -DHAVE_AVX512BW is not defined, -DCOMPILE_AVX512BW has no effect.+ Likewise, if -DHAVE_AVX2 is not defined, -DCOMPILE_AVX2 has no effect.+*/++#if defined(COMPILE_AVX512BW)+/// Compile an optimized AVX512BW version defined in matcher_avx2.cpp+size_t Matcher::simd_match_avx512bw(Method method) {- // Relative frequency table of English letters, source code, and UTF-8 bytes- static unsigned char freq[256] = "\0\0\0\0\0\0\0\0\0\73\4\0\0\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\73\70\70\1\1\2\2\70\70\70\2\2\70\70\70\2\3\3\3\3\3\3\3\3\3\3\70\70\70\70\70\70\2\35\14\24\26\37\20\17\30\33\11\12\25\22\32\34\15\7\27\31\36\23\13\21\10\16\6\70\1\70\2\70\1\67\46\56\60\72\52\51\62\65\43\44\57\54\64\66\47\41\61\63\71\55\45\53\42\50\40\70\2\70\2\0\47\47\47\47\47\47\47\47\47\47\47\47\47\47\47\47\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\44\44\44\44\44\44\44\44\44\44\44\44\44\44\44\44\0\0\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\46\56\56\56\56\56\56\56\56\56\56\56\56\46\56\56\73\0\0\0\0\0\0\0\0\0\0\0\0\0\0";- uint8_t n = static_cast<uint8_t>(len); // okay to cast: actually never more than 255- uint16_t i;- for (i = 0; i < 256; ++i)- bms_[i] = n;- lcp_ = 0;- lcs_ = n > 1;- for (i = 0; i < n; ++i)+#elif defined(COMPILE_AVX2)+/// Compile an optimized AVX2 version defined in matcher_avx512bw.cpp+size_t Matcher::simd_match_avx2(Method method)+{+#else+/// Returns true if input matched the pattern using method Const::SCAN, Const::FIND, Const::SPLIT, or Const::MATCH.+size_t Matcher::match(Method method)+{+ DBGLOG("BEGIN Matcher::match()");+#if defined(HAVE_AVX512BW) && (!defined(_MSC_VER) || defined(_WIN64))+ if (have_HW_AVX512BW())+ return simd_match_avx512bw(method);+ if (have_HW_AVX2())+ return simd_match_avx2(method);+#elif defined(HAVE_AVX2)+ if (have_HW_AVX2())+ return simd_match_avx2(method);+#endif+#endif+ reset_text();+ len_ = 0; // split text length starts with 0+ anc_ = false; // no word boundary anchor found and applied+scan:+ txt_ = buf_ + cur_;+#if !defined(WITH_NO_INDENT)+ mrk_ = false;+ ind_ = pos_; // ind scans input in buf[] in newline() up to pos - 1+ col_ = 0; // count columns for indent matching+#endif+find:+ int c1 = got_;+ bool bol = at_bol(); // at begin of line?+ if (pat_->fsm_ != NULL)+ fsm_.c1 = c1;+#if !defined(WITH_NO_INDENT)+redo:+#endif+ lap_.resize(0);+ cap_ = 0;+ bool nul = method == Const::MATCH;+ if (pat_->fsm_ != NULL) {- uint8_t pch = static_cast<uint8_t>(pat[i]);- bms_[pch] = static_cast<uint8_t>(n - i - 1);- if (i > 0)+ DBGLOG("FSM code %p", pat_->fsm_);+ fsm_.bol = bol;+ fsm_.nul = nul;+ pat_->fsm_(*this);+ nul = fsm_.nul;+ c1 = fsm_.c1;+ }+ else if (pat_->opc_ != NULL)+ {+ const Pattern::Opcode *pc = pat_->opc_;+ while (true) {- if (freq[static_cast<uint8_t>(pat[lcp_])] > freq[pch])+ Pattern::Opcode opcode = *pc;+ DBGLOG("Fetch: code[%zu] = 0x%08X", pc - pat_->opc_, opcode);+ if (!Pattern::is_opcode_goto(opcode)) {- lcs_ = lcp_;- lcp_ = i;+ switch (opcode >> 24)+ {+ case 0xFE: // TAKE+ cap_ = Pattern::long_index_of(opcode);+ cur_ = pos_;+ ++pc;+ DBGLOG("Take: cap = %zu", cap_);+ continue;+ case 0xFD: // REDO+ cap_ = Const::REDO;+ DBGLOG("Redo");+ cur_ = pos_;+ ++pc;+ continue;+ case 0xFC: // TAIL+ {+ Pattern::Lookahead la = Pattern::lookahead_of(opcode);+ DBGLOG("Tail: %u", la);+ if (lap_.size() > la && lap_[la] >= 0)+ cur_ = txt_ - buf_ + static_cast<size_t>(lap_[la]); // mind the (new) gap+ ++pc;+ continue;+ }+ case 0xFB: // HEAD+ {+ Pattern::Lookahead la = Pattern::lookahead_of(opcode);+ DBGLOG("Head: lookahead[%u] = %zu", la, pos_ - (txt_ - buf_));+ if (lap_.size() <= la)+ lap_.resize(la + 1, -1);+ lap_[la] = static_cast<int>(pos_ - (txt_ - buf_)); // mind the gap+ ++pc;+ continue;+ }+#if !defined(WITH_NO_INDENT)+ case Pattern::META_DED - Pattern::META_MIN:+ if (ded_ > 0)+ {+ Pattern::Index jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(pc[1]);+ DBGLOG("Dedent ded = %zu", ded_); // unconditional dedent matching \j+ nul = true;+ pc = pat_->opc_ + jump;+ continue;+ }+#endif+ }+ if (c1 == EOF)+ break;+ int c0 = c1;+ c1 = get();+ DBGLOG("Get: c1 = %d", c1);+ Pattern::Index back = Pattern::Const::IMAX; // where to jump back to (backtrack on meta transitions)+ Pattern::Index jump = Pattern::Const::IMAX; // to jump to longest sequence of matching metas+ while (true)+ {+ if ((jump == Pattern::Const::IMAX || back == Pattern::Const::IMAX) && !Pattern::is_opcode_goto(opcode))+ {+ // we no longer have to pass through all if jump and back are set+ switch (opcode >> 24)+ {+ case 0xFE: // TAKE+ cap_ = Pattern::long_index_of(opcode);+ cur_ = pos_;+ if (c1 != EOF)+ --cur_; // must unget one char+ opcode = *++pc;+ DBGLOG("Take: cap = %zu", cap_);+ continue;+ case 0xFD: // REDO+ cap_ = Const::REDO;+ DBGLOG("Redo");+ cur_ = pos_;+ if (c1 != EOF)+ --cur_; // must unget one char+ opcode = *++pc;+ continue;+ case 0xFC: // TAIL+ {+ Pattern::Lookahead la = Pattern::lookahead_of(opcode);+ DBGLOG("Tail: %u", la);+ if (lap_.size() > la && lap_[la] >= 0)+ cur_ = txt_ - buf_ + static_cast<size_t>(lap_[la]); // mind the (new) gap+ opcode = *++pc;+ continue;+ }+ case 0xFB: // HEAD+ opcode = *++pc;+ continue;+#if !defined(WITH_NO_INDENT)+ case Pattern::META_DED - Pattern::META_MIN:+ DBGLOG("DED? %d", c1);+ if (jump == Pattern::Const::IMAX && back == Pattern::Const::IMAX && bol && dedent())+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case Pattern::META_IND - Pattern::META_MIN:+ DBGLOG("IND? %d", c1);+ if (jump == Pattern::Const::IMAX && back == Pattern::Const::IMAX && bol && indent())+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case Pattern::META_UND - Pattern::META_MIN:+ DBGLOG("UND");+ if (mrk_)+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ mrk_ = false;+ ded_ = 0;+ opcode = *++pc;+ continue;+#endif+ case Pattern::META_EOB - Pattern::META_MIN:+ DBGLOG("EOB? %d", c1);+ if (jump == Pattern::Const::IMAX && c1 == EOF)+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case Pattern::META_BOB - Pattern::META_MIN:+ DBGLOG("BOB? %d", at_bob());+ if (jump == Pattern::Const::IMAX && at_bob())+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case Pattern::META_EOL - Pattern::META_MIN:+ DBGLOG("EOL? %d", c1);+ anc_ = true;+ if (jump == Pattern::Const::IMAX && (c1 == EOF || c1 == '\n' || (c1 == '\r' && peek() == '\n')))+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case Pattern::META_BOL - Pattern::META_MIN:+ DBGLOG("BOL? %d", bol);+ anc_ = true;+ if (jump == Pattern::Const::IMAX && bol)+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case Pattern::META_EWE - Pattern::META_MIN:+ DBGLOG("EWE? %d %d %d", c0, c1, isword(c0) && !isword(c1));+ anc_ = true;+ if (jump == Pattern::Const::IMAX && (isword(c0) || opt_.W) && !isword(c1))+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case Pattern::META_BWE - Pattern::META_MIN:+ DBGLOG("BWE? %d %d %d", c0, c1, !isword(c0) && isword(c1));+ anc_ = true;+ if (jump == Pattern::Const::IMAX && !isword(c0) && isword(c1))+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case Pattern::META_EWB - Pattern::META_MIN:+ DBGLOG("EWB? %d", at_eow());+ anc_ = true;+ if (jump == Pattern::Const::IMAX && isword(got_) &&+ !isword(static_cast<unsigned char>(txt_[len_])))+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case Pattern::META_BWB - Pattern::META_MIN:+ DBGLOG("BWB? %d", at_bow());+ anc_ = true;+ if (jump == Pattern::Const::IMAX && !isword(got_) &&+ (opt_.W || isword(static_cast<unsigned char>(txt_[len_]))))+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case Pattern::META_NWE - Pattern::META_MIN:+ DBGLOG("NWE? %d %d %d", c0, c1, isword(c0) == isword(c1));+ anc_ = true;+ if (jump == Pattern::Const::IMAX && isword(c0) == isword(c1))+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case Pattern::META_NWB - Pattern::META_MIN:+ DBGLOG("NWB? %d %d", at_bow(), at_eow());+ anc_ = true;+ if (jump == Pattern::Const::IMAX &&+ isword(got_) == isword(static_cast<unsigned char>(txt_[len_])))+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case 0xFF: // LONG+ opcode = *++pc;+ continue;+ }+ }+ if (jump == Pattern::Const::IMAX)+ {+ if (back != Pattern::Const::IMAX)+ {+ pc = pat_->opc_ + back;+ opcode = *pc;+ }+ break;+ }+ DBGLOG("Backtrack: pc = %u", jump);+ if (back == Pattern::Const::IMAX)+ back = static_cast<Pattern::Index>(pc - pat_->opc_);+ pc = pat_->opc_ + jump;+ opcode = *pc;+ jump = Pattern::Const::IMAX;+ }+ if (c1 == EOF)+ break; }- else if (freq[static_cast<uint8_t>(pat[lcs_])] > freq[pch])+ else {- lcs_ = i;+ if (Pattern::is_opcode_halt(opcode))+ break;+ if (c1 == EOF)+ break;+ c1 = get();+ DBGLOG("Get: c1 = %d", c1);+ if (c1 == EOF)+ break; }+ Pattern::Opcode lo = c1 << 24;+ Pattern::Opcode hi = lo | 0x00FFFFFF;+unrolled:+ if (hi < opcode || lo > (opcode << 8))+ {+ opcode = *++pc;+ if (hi < opcode || lo > (opcode << 8))+ {+ opcode = *++pc;+ if (hi < opcode || lo > (opcode << 8))+ {+ opcode = *++pc;+ if (hi < opcode || lo > (opcode << 8))+ {+ opcode = *++pc;+ if (hi < opcode || lo > (opcode << 8))+ {+ opcode = *++pc;+ if (hi < opcode || lo > (opcode << 8))+ {+ opcode = *++pc;+ if (hi < opcode || lo > (opcode << 8))+ {+ opcode = *++pc;+ if (hi < opcode || lo > (opcode << 8))+ {+ opcode = *++pc;+ goto unrolled;+ }+ }+ }+ }+ }+ }+ }+ }+ Pattern::Index jump = Pattern::index_of(opcode);+ if (jump == 0)+ {+ // loop back to start state after only one char matched (one transition) but w/o full match, then optimize+ if (cap_ == 0 && pos_ == cur_ + 1 && method == Const::FIND)+ cur_ = pos_; // set cur_ to move forward from cur_ + 1 with FIND advance()+ }+ else if (jump >= Pattern::Const::LONG)+ {+ if (jump == Pattern::Const::HALT)+ break;+ jump = Pattern::long_index_of(pc[1]);+ }+ pc = pat_->opc_ + jump; } }- uint16_t j;- for (i = n - 1, j = i; j > 0; --j)- if (pat[j - 1] == pat[i])- break;- bmd_ = i - j + 1;-#if defined(HAVE_AVX512BW) || defined(HAVE_AVX2) || defined(HAVE_SSE2) || defined(__SSE2__) || defined(__x86_64__) || _M_IX86_FP == 2- size_t score = 0;- for (i = 0; i < n; ++i)- score += bms_[static_cast<uint8_t>(pat[i])];- score /= n;- uint8_t fch = freq[static_cast<uint8_t>(pat[lcp_])];-#if defined(HAVE_AVX512BW) || defined(HAVE_AVX2) || defined(HAVE_SSE2)- if (!have_HW_SSE2() && !have_HW_AVX2() && !have_HW_AVX512BW())+#if !defined(WITH_NO_INDENT)+ if (mrk_ && cap_ != Const::REDO) {- // if scoring is high and freq is high, then use our improved Boyer-Moore instead of memchr()-#if defined(__SSE2__) || defined(__x86_64__) || _M_IX86_FP == 2- // SSE2 is available, expect fast memchr() to use instead of BM- if (score > 1 && fch > 35 && (score > 3 || fch > 50) && fch + score > 52)- lcs_ = 0xffff;-#else- // no SSE2 available, expect slow memchr() and use BM unless score or frequency are too low- if (fch > 37 || (fch > 8 && score > 0))- lcs_ = 0xffff;-#endif+ if (col_ > 0 && (tab_.empty() || tab_.back() < col_))+ {+ DBGLOG("Set new stop: tab_[%zu] = %zu", tab_.size(), col_);+ tab_.push_back(col_);+ }+ else if (!tab_.empty() && tab_.back() > col_)+ {+ size_t n;+ for (n = tab_.size() - 1; n > 0; --n)+ if (tab_.at(n - 1) <= col_)+ break;+ ded_ += tab_.size() - n;+ DBGLOG("Dedents: ded = %zu tab_ = %zu", ded_, tab_.size());+ tab_.resize(n);+ // adjust stop when indents are not aligned (Python would give an error though)+ if (n > 0)+ tab_.back() = col_;+ }+ }+ if (ded_ > 0)+ {+ DBGLOG("Dedents: ded = %zu", ded_);+ if (col_ == 0 && bol)+ {+ ded_ += tab_.size();+ tab_.resize(0);+ DBGLOG("Rescan for pending dedents: ded = %zu", ded_);+ pos_ = ind_;+ // avoid looping, match \j exactly+ bol = false;+ goto redo;+ }+ --ded_; }-#elif defined(__SSE2__) || defined(__x86_64__) || _M_IX86_FP == 2- // SSE2 is available, expect fast memchr() to use instead of BM- if (score > 1 && fch > 35 && (score > 3 || fch > 50) && fch + score > 52)- lcs_ = 0xffff; #endif+ if (method == Const::SPLIT)+ {+ DBGLOG("Split: len = %zu cap = %zu cur = %zu pos = %zu end = %zu txt-buf = %zu eob = %d got = %d", len_, cap_, cur_, pos_, end_, txt_-buf_, (int)eof_, got_);+ if (cap_ == 0 || (cur_ == static_cast<size_t>(txt_ - buf_) && !at_bob()))+ {+ if (!hit_end() && (txt_ + len_ < buf_ + end_ || peek() != EOF))+ {+ ++len_;+ DBGLOG("Split continue: len = %zu", len_);+ set_current(++cur_);+ goto find;+ }+ if (got_ != Const::EOB)+ cap_ = Const::EMPTY;+ else+ cap_ = 0;+ set_current(end_);+ got_ = Const::EOB;+ DBGLOG("Split at eof: cap = %zu txt = '%s' len = %zu", cap_, std::string(txt_, len_).c_str(), len_);+ DBGLOG("END Matcher::match()");+ return cap_;+ }+ if (cur_ == 0 && at_bob() && at_end())+ {+ cap_ = Const::EMPTY;+ got_ = Const::EOB;+ }+ else+ {+ set_current(cur_);+ }+ DBGLOG("Split: txt = '%s' len = %zu", std::string(txt_, len_).c_str(), len_);+ DBGLOG("END Matcher::match()");+ return cap_;+ }+ if (cap_ == 0)+ {+ if (method == Const::FIND)+ {+ if (!at_end())+ {+ if (anc_)+ {+ cur_ = txt_ - buf_; // reset current to pattern start when a word boundary was encountered+ anc_ = false;+ }+ if (pos_ > cur_) // if we didn't fail on META alone+ {+ if (+#if defined(COMPILE_AVX512BW)+ simd_advance_avx512bw()+#elif defined(COMPILE_AVX2)+ simd_advance_avx2()+#else+ advance() #endif+ )+ {+ if (!pat_->one_)+ goto scan;+ txt_ = buf_ + cur_;+ len_ = pat_->len_;+ set_current(cur_ + len_);+ return cap_ = 1;+ }+ }+ txt_ = buf_ + cur_;+ }+ }+ else+ {+ // SCAN and MATCH: no match: backup to begin of unmatched text to report as error+ cur_ = txt_ - buf_;+ }+ }+ len_ = cur_ - (txt_ - buf_);+ if (len_ == 0 && !nul)+ {+ DBGLOG("Empty or no match cur = %zu pos = %zu end = %zu", cur_, pos_, end_);+ pos_ = cur_;+ if (at_end())+ {+ set_current(cur_);+ DBGLOG("Reject empty match at EOF");+ cap_ = 0;+ }+ else if (method == Const::FIND)+ {+ DBGLOG("Reject empty match and continue?");+ // skip one char to keep searching+ set_current(++cur_);+ // allow FIND with "N" to match an empty line, with ^$ etc.+ if (cap_ == 0 || !opt_.N)+ goto scan;+ DBGLOG("Accept empty match");+ }+ else+ {+ set_current(cur_);+ DBGLOG("Reject empty match");+ cap_ = 0;+ }+ }+ else if (len_ == 0 && cur_ == end_)+ {+ DBGLOG("Hit end: got = %d", got_);+ if (cap_ == Const::REDO && !opt_.A)+ cap_ = 0;+ }+ else+ {+ set_current(cur_);+ if (len_ > 0 && cap_ == Const::REDO && !opt_.A)+ {+ DBGLOG("Ignore accept and continue: len = %zu", len_);+ len_ = 0;+ if (method != Const::MATCH)+ goto scan;+ cap_ = 0;+ }+ }+ DBGLOG("Return: cap = %zu txt = '%s' len = %zu pos = %zu got = %d", cap_, std::string(txt_, len_).c_str(), len_, pos_, got_);+ DBGLOG("END match()");+ return cap_; } -// advance input cursor position after mismatch to align input for the next match+#if defined(COMPILE_AVX512BW)+/// Compile an optimized AVX512BW version defined in matcher_avx512bw.cpp+bool Matcher::simd_advance_avx512bw()+{+#elif defined(COMPILE_AVX2)+/// Compile an optimized AVX2 version defined in matcher_avx2.cpp+bool Matcher::simd_advance_avx2()+{+#else+/// advance input cursor position after mismatch to align input for the next match bool Matcher::advance() {+#endif size_t loc = cur_ + 1; size_t min = pat_->min_; if (pat_->len_ == 0)@@ -286,141 +810,173 @@ bool Matcher::advance() } } if (bmd_ == 0)- boyer_moore_init(pre, len);+ {+ // Boyer-Moore preprocessing of the given pattern pat of length len, generates bmd_ > 0 and bms_[] shifts.+ // relative frequency table of English letters, source code, and UTF-8 bytes+ static unsigned char freq[256] = "\0\0\0\0\0\0\0\0\0\73\4\0\0\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\73\70\70\1\1\2\2\70\70\70\2\2\70\70\70\2\3\3\3\3\3\3\3\3\3\3\70\70\70\70\70\70\2\35\14\24\26\37\20\17\30\33\11\12\25\22\32\34\15\7\27\31\36\23\13\21\10\16\6\70\1\70\2\70\1\67\46\56\60\72\52\51\62\65\43\44\57\54\64\66\47\41\61\63\71\55\45\53\42\50\40\70\2\70\2\0\47\47\47\47\47\47\47\47\47\47\47\47\47\47\47\47\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\44\44\44\44\44\44\44\44\44\44\44\44\44\44\44\44\0\0\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\46\56\56\56\56\56\56\56\56\56\56\56\56\46\56\56\73\0\0\0\0\0\0\0\0\0\0\0\0\0\0";+ uint8_t n = static_cast<uint8_t>(len); // okay to cast: actually never more than 255+ uint16_t i;+ for (i = 0; i < 256; ++i)+ bms_[i] = n;+ lcp_ = 0;+ lcs_ = n > 1;+ for (i = 0; i < n; ++i)+ {+ uint8_t pch = static_cast<uint8_t>(pre[i]);+ bms_[pch] = static_cast<uint8_t>(n - i - 1);+ if (i > 0)+ {+ if (freq[static_cast<uint8_t>(pre[lcp_])] > freq[pch])+ {+ lcs_ = lcp_;+ lcp_ = i;+ }+ else if (freq[static_cast<uint8_t>(pre[lcs_])] > freq[pch])+ {+ lcs_ = i;+ }+ }+ }+ uint16_t j;+ for (i = n - 1, j = i; j > 0; --j)+ if (pre[j - 1] == pre[i])+ break;+ bmd_ = i - j + 1;+#if defined(HAVE_AVX512BW) || defined(HAVE_AVX2) || defined(HAVE_SSE2) || defined(__SSE2__) || defined(__x86_64__) || _M_IX86_FP == 2+ size_t score = 0;+ for (i = 0; i < n; ++i)+ score += bms_[static_cast<uint8_t>(pre[i])];+ score /= n;+ uint8_t fch = freq[static_cast<uint8_t>(pre[lcp_])];+#if defined(HAVE_AVX512BW) || defined(HAVE_AVX2) || defined(HAVE_SSE2)+ if (!have_HW_SSE2() && !have_HW_AVX2() && !have_HW_AVX512BW())+ {+ // if scoring is high and freq is high, then use our improved Boyer-Moore instead of memchr()+#if defined(__SSE2__) || defined(__x86_64__) || _M_IX86_FP == 2+ // SSE2 is available, expect fast memchr() to use instead of BM+ if (score > 1 && fch > 35 && (score > 3 || fch > 50) && fch + score > 52)+ lcs_ = 0xffff;+#else+ // no SSE2 available, expect slow memchr() and use BM unless score or frequency are too low+ if (fch > 37 || (fch > 8 && score > 0))+ lcs_ = 0xffff;+#endif+ }+#elif defined(__SSE2__) || defined(__x86_64__) || _M_IX86_FP == 2+ // SSE2 is available, expect fast memchr() to use instead of BM+ if (score > 1 && fch > 35 && (score > 3 || fch > 50) && fch + score > 52)+ lcs_ = 0xffff;+#endif+#endif+ } while (true) { if (lcs_ < len) { const char *s = buf_ + loc + lcp_; const char *e = buf_ + end_ + lcp_ - len + 1;-#if defined(HAVE_AVX512BW) && (!defined(_MSC_VER) || defined(_WIN64))- if (s + 64 > e && have_HW_AVX512BW())+#if defined(COMPILE_AVX512BW)+ // implements AVX512BW string search scheme based on in http://0x80.pl/articles/simd-friendly-karp-rabin.html+ __m512i vlcp = _mm512_set1_epi8(pre[lcp_]);+ __m512i vlcs = _mm512_set1_epi8(pre[lcs_]);+ while (s + 64 <= e) {- if (simd_advance_avx512bw(s, e, loc, min, pre, len))- return true;- }- else if (s + 32 > e && have_HW_AVX2())- {- if (simd_advance_avx2(s, e, loc, min, pre, len))- return true;- }- else if (have_HW_SSE2())- {- // implements SSE2 string search scheme based on in http://0x80.pl/articles/simd-friendly-karp-rabin.html- __m128i vlcp = _mm_set1_epi8(pre[lcp_]);- __m128i vlcs = _mm_set1_epi8(pre[lcs_]);- while (s + 16 <= e)+ __m512i vlcpm = _mm512_loadu_si512(reinterpret_cast<const __m512i*>(s));+ __m512i vlcsm = _mm512_loadu_si512(reinterpret_cast<const __m512i*>(s + lcs_ - lcp_));+ uint64_t mask = _mm512_cmpeq_epi8_mask(vlcp, vlcpm) & _mm512_cmpeq_epi8_mask(vlcs, vlcsm);+ while (mask != 0) {- __m128i vlcpm = _mm_loadu_si128(reinterpret_cast<const __m128i*>(s));- __m128i vlcsm = _mm_loadu_si128(reinterpret_cast<const __m128i*>(s + lcs_ - lcp_));- __m128i vlcpeq = _mm_cmpeq_epi8(vlcp, vlcpm);- __m128i vlcseq = _mm_cmpeq_epi8(vlcs, vlcsm);- uint32_t mask = _mm_movemask_epi8(_mm_and_si128(vlcpeq, vlcseq));- while (mask != 0)+ uint32_t offset = ctzl(mask);+ if (std::memcmp(s - lcp_ + offset, pre, len) == 0) {- uint32_t offset = ctz(mask);- if (std::memcmp(s - lcp_ + offset, pre, len) == 0)+ loc = s - lcp_ + offset - buf_;+ set_current(loc);+ if (min == 0)+ return true;+ if (min >= 4) {- loc = s - lcp_ + offset - buf_;- set_current(loc);- if (min == 0)+ if (loc + len + min > end_ || Pattern::predict_match(pat_->pmh_, &buf_[loc + len], min))+ return true;+ }+ else+ {+ if (loc + len + 4 > end_ || Pattern::predict_match(pat_->pma_, &buf_[loc + len]) == 0) return true;- if (min >= 4)- {- if (loc + len + min > end_ || Pattern::predict_match(pat_->pmh_, &buf_[loc + len], min))- return true;- }- else- {- if (loc + len + 4 > end_ || Pattern::predict_match(pat_->pma_, &buf_[loc + len]) == 0)- return true;- } }- mask &= mask - 1; }- s += 16;+ mask &= mask - 1; }+ s += 64; }-#elif defined(HAVE_AVX2)- if (s + 32 > e && have_HW_AVX2())+#elif defined(COMPILE_AVX2)+ // implements AVX2 string search scheme based on in http://0x80.pl/articles/simd-friendly-karp-rabin.html+ __m256i vlcp = _mm256_set1_epi8(pre[lcp_]);+ __m256i vlcs = _mm256_set1_epi8(pre[lcs_]);+ while (s + 32 <= e) {- if (simd_advance_avx2(s, e, loc, min, pre, len))- return true;- }- else if (have_HW_SSE2())- {- // implements SSE2 string search scheme based on in http://0x80.pl/articles/simd-friendly-karp-rabin.html- __m128i vlcp = _mm_set1_epi8(pre[lcp_]);- __m128i vlcs = _mm_set1_epi8(pre[lcs_]);- while (s + 16 <= e)+ __m256i vlcpm = _mm256_loadu_si256(reinterpret_cast<const __m256i*>(s));+ __m256i vlcsm = _mm256_loadu_si256(reinterpret_cast<const __m256i*>(s + lcs_ - lcp_));+ __m256i vlcpeq = _mm256_cmpeq_epi8(vlcp, vlcpm);+ __m256i vlcseq = _mm256_cmpeq_epi8(vlcs, vlcsm);+ uint32_t mask = _mm256_movemask_epi8(_mm256_and_si256(vlcpeq, vlcseq));+ while (mask != 0) {- __m128i vlcpm = _mm_loadu_si128(reinterpret_cast<const __m128i*>(s));- __m128i vlcsm = _mm_loadu_si128(reinterpret_cast<const __m128i*>(s + lcs_ - lcp_));- __m128i vlcpeq = _mm_cmpeq_epi8(vlcp, vlcpm);- __m128i vlcseq = _mm_cmpeq_epi8(vlcs, vlcsm);- uint32_t mask = _mm_movemask_epi8(_mm_and_si128(vlcpeq, vlcseq));- while (mask != 0)+ uint32_t offset = ctz(mask);+ if (std::memcmp(s - lcp_ + offset, pre, len) == 0) {- uint32_t offset = ctz(mask);- if (std::memcmp(s - lcp_ + offset, pre, len) == 0)+ loc = s - lcp_ + offset - buf_;+ set_current(loc);+ if (min == 0)+ return true;+ if (min >= 4) {- loc = s - lcp_ + offset - buf_;- set_current(loc);- if (min == 0)+ if (loc + len + min > end_ || Pattern::predict_match(pat_->pmh_, &buf_[loc + len], min))+ return true;+ }+ else+ {+ if (loc + len + 4 > end_ || Pattern::predict_match(pat_->pma_, &buf_[loc + len]) == 0) return true;- if (min >= 4)- {- if (loc + len + min > end_ || Pattern::predict_match(pat_->pmh_, &buf_[loc + len], min))- return true;- }- else- {- if (loc + len + 4 > end_ || Pattern::predict_match(pat_->pma_, &buf_[loc + len]) == 0)- return true;- } }- mask &= mask - 1; }- s += 16;+ mask &= mask - 1; }+ s += 32; } #elif defined(HAVE_SSE2)- if (have_HW_SSE2())+ // implements SSE2 string search scheme based on in http://0x80.pl/articles/simd-friendly-karp-rabin.html+ __m128i vlcp = _mm_set1_epi8(pre[lcp_]);+ __m128i vlcs = _mm_set1_epi8(pre[lcs_]);+ while (s + 16 <= e) {- // implements SSE2 string search scheme based on in http://0x80.pl/articles/simd-friendly-karp-rabin.html- __m128i vlcp = _mm_set1_epi8(pre[lcp_]);- __m128i vlcs = _mm_set1_epi8(pre[lcs_]);- while (s + 16 <= e)+ __m128i vlcpm = _mm_loadu_si128(reinterpret_cast<const __m128i*>(s));+ __m128i vlcsm = _mm_loadu_si128(reinterpret_cast<const __m128i*>(s + lcs_ - lcp_));+ __m128i vlcpeq = _mm_cmpeq_epi8(vlcp, vlcpm);+ __m128i vlcseq = _mm_cmpeq_epi8(vlcs, vlcsm);+ uint32_t mask = _mm_movemask_epi8(_mm_and_si128(vlcpeq, vlcseq));+ while (mask != 0) {- __m128i vlcpm = _mm_loadu_si128(reinterpret_cast<const __m128i*>(s));- __m128i vlcsm = _mm_loadu_si128(reinterpret_cast<const __m128i*>(s + lcs_ - lcp_));- __m128i vlcpeq = _mm_cmpeq_epi8(vlcp, vlcpm);- __m128i vlcseq = _mm_cmpeq_epi8(vlcs, vlcsm);- uint32_t mask = _mm_movemask_epi8(_mm_and_si128(vlcpeq, vlcseq));- while (mask != 0)+ uint32_t offset = ctz(mask);+ if (std::memcmp(s - lcp_ + offset, pre, len) == 0) {- uint32_t offset = ctz(mask);- if (std::memcmp(s - lcp_ + offset, pre, len) == 0)+ loc = s - lcp_ + offset - buf_;+ set_current(loc);+ if (min == 0)+ return true;+ if (min >= 4) {- loc = s - lcp_ + offset - buf_;- set_current(loc);- if (min == 0)+ if (loc + len + min > end_ || Pattern::predict_match(pat_->pmh_, &buf_[loc + len], min))+ return true;+ }+ else+ {+ if (loc + len + 4 > end_ || Pattern::predict_match(pat_->pma_, &buf_[loc + len]) == 0) return true;- if (min >= 4)- {- if (loc + len + min > end_ || Pattern::predict_match(pat_->pmh_, &buf_[loc + len], min))- return true;- }- else- {- if (loc + len + 4 > end_ || Pattern::predict_match(pat_->pma_, &buf_[loc + len]) == 0)- return true;- } }- mask &= mask - 1; }- s += 16;+ mask &= mask - 1; }+ s += 16; } #elif defined(HAVE_NEON) // implements NEON/AArch64 string search scheme based on in http://0x80.pl/articles/simd-friendly-karp-rabin.html but 64 bit optimized@@ -586,3 +1142,5 @@ bool Matcher::advance() } } // namespace reflex++#endif
@@ -0,0 +1,1146 @@+/******************************************************************************\+* Copyright (c) 2016, Robert van Engelen, Genivia Inc. All rights reserved. *+* *+* Redistribution and use in source and binary forms, with or without *+* modification, are permitted provided that the following conditions are met: *+* *+* (1) Redistributions of source code must retain the above copyright notice, *+* this list of conditions and the following disclaimer. *+* *+* (2) Redistributions in binary form must reproduce the above copyright *+* notice, this list of conditions and the following disclaimer in the *+* documentation and/or other materials provided with the distribution. *+* *+* (3) The name of the author may not be used to endorse or promote products *+* derived from this software without specific prior written permission. *+* *+* THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR IMPLIED *+* WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF *+* MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO *+* EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, *+* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, *+* PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; *+* OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, *+* WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR *+* OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF *+* ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. *+\******************************************************************************/++/**+@file matcher.cpp, matcher_avx2.cpp, matcher_avx512bw.cpp (same file)+@brief RE/flex matcher engine+@author Robert van Engelen - engelen@genivia.com+@copyright (c) 2016-2022, Robert van Engelen, Genivia Inc. All rights reserved.+@copyright (c) BSD-3 License - see LICENSE.txt+*/++#if defined(COMPILE_AVX512BW) && !defined(HAVE_AVX512BW)++// appease ranlib "has no symbols"+void matcher_not_compiled_with_avx512bw() { }++#elif defined(COMPILE_AVX2) && !defined(HAVE_AVX2)++// appease ranlib "has no symbols"+void matcher_not_compiled_with_avx2() { }++#else++#include <reflex/matcher.h>++namespace reflex {++/*+ The simd_match_avx512bw() and simd_match_avx2() methods are AVX-optimized+ versions of the match() method. To compile these methods separately with+ the appropriate compilation flags, this file is copied to+ matcher_avx512bw.cpp and matcher_avx2.cpp then compiled with -mavx512bw+ -DCOMPILE_AVX512BW -DHAVE_AVX512BW and with -mavx2 -DCOMPILE_AVX2+ -DHAVE_AVX2, respectively. Likewise, the simd_advance_avx512bw() and+ simd_advance_avx2() methods are optimized versions and separately compiled.+ This approach is preferred over maintaining three separate copies of source+ code files with these methods that only slightly differ. On the other hand,+ combining these versions into one source file means more #if branches.++ If -DHAVE_AVX512BW is not defined, -DCOMPILE_AVX512BW has no effect.+ Likewise, if -DHAVE_AVX2 is not defined, -DCOMPILE_AVX2 has no effect.+*/++#if defined(COMPILE_AVX512BW)+/// Compile an optimized AVX512BW version defined in matcher_avx2.cpp+size_t Matcher::simd_match_avx512bw(Method method)+{+#elif defined(COMPILE_AVX2)+/// Compile an optimized AVX2 version defined in matcher_avx512bw.cpp+size_t Matcher::simd_match_avx2(Method method)+{+#else+/// Returns true if input matched the pattern using method Const::SCAN, Const::FIND, Const::SPLIT, or Const::MATCH.+size_t Matcher::match(Method method)+{+ DBGLOG("BEGIN Matcher::match()");+#if defined(HAVE_AVX512BW) && (!defined(_MSC_VER) || defined(_WIN64))+ if (have_HW_AVX512BW())+ return simd_match_avx512bw(method);+ if (have_HW_AVX2())+ return simd_match_avx2(method);+#elif defined(HAVE_AVX2)+ if (have_HW_AVX2())+ return simd_match_avx2(method);+#endif+#endif+ reset_text();+ len_ = 0; // split text length starts with 0+ anc_ = false; // no word boundary anchor found and applied+scan:+ txt_ = buf_ + cur_;+#if !defined(WITH_NO_INDENT)+ mrk_ = false;+ ind_ = pos_; // ind scans input in buf[] in newline() up to pos - 1+ col_ = 0; // count columns for indent matching+#endif+find:+ int c1 = got_;+ bool bol = at_bol(); // at begin of line?+ if (pat_->fsm_ != NULL)+ fsm_.c1 = c1;+#if !defined(WITH_NO_INDENT)+redo:+#endif+ lap_.resize(0);+ cap_ = 0;+ bool nul = method == Const::MATCH;+ if (pat_->fsm_ != NULL)+ {+ DBGLOG("FSM code %p", pat_->fsm_);+ fsm_.bol = bol;+ fsm_.nul = nul;+ pat_->fsm_(*this);+ nul = fsm_.nul;+ c1 = fsm_.c1;+ }+ else if (pat_->opc_ != NULL)+ {+ const Pattern::Opcode *pc = pat_->opc_;+ while (true)+ {+ Pattern::Opcode opcode = *pc;+ DBGLOG("Fetch: code[%zu] = 0x%08X", pc - pat_->opc_, opcode);+ if (!Pattern::is_opcode_goto(opcode))+ {+ switch (opcode >> 24)+ {+ case 0xFE: // TAKE+ cap_ = Pattern::long_index_of(opcode);+ cur_ = pos_;+ ++pc;+ DBGLOG("Take: cap = %zu", cap_);+ continue;+ case 0xFD: // REDO+ cap_ = Const::REDO;+ DBGLOG("Redo");+ cur_ = pos_;+ ++pc;+ continue;+ case 0xFC: // TAIL+ {+ Pattern::Lookahead la = Pattern::lookahead_of(opcode);+ DBGLOG("Tail: %u", la);+ if (lap_.size() > la && lap_[la] >= 0)+ cur_ = txt_ - buf_ + static_cast<size_t>(lap_[la]); // mind the (new) gap+ ++pc;+ continue;+ }+ case 0xFB: // HEAD+ {+ Pattern::Lookahead la = Pattern::lookahead_of(opcode);+ DBGLOG("Head: lookahead[%u] = %zu", la, pos_ - (txt_ - buf_));+ if (lap_.size() <= la)+ lap_.resize(la + 1, -1);+ lap_[la] = static_cast<int>(pos_ - (txt_ - buf_)); // mind the gap+ ++pc;+ continue;+ }+#if !defined(WITH_NO_INDENT)+ case Pattern::META_DED - Pattern::META_MIN:+ if (ded_ > 0)+ {+ Pattern::Index jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(pc[1]);+ DBGLOG("Dedent ded = %zu", ded_); // unconditional dedent matching \j+ nul = true;+ pc = pat_->opc_ + jump;+ continue;+ }+#endif+ }+ if (c1 == EOF)+ break;+ int c0 = c1;+ c1 = get();+ DBGLOG("Get: c1 = %d", c1);+ Pattern::Index back = Pattern::Const::IMAX; // where to jump back to (backtrack on meta transitions)+ Pattern::Index jump = Pattern::Const::IMAX; // to jump to longest sequence of matching metas+ while (true)+ {+ if ((jump == Pattern::Const::IMAX || back == Pattern::Const::IMAX) && !Pattern::is_opcode_goto(opcode))+ {+ // we no longer have to pass through all if jump and back are set+ switch (opcode >> 24)+ {+ case 0xFE: // TAKE+ cap_ = Pattern::long_index_of(opcode);+ cur_ = pos_;+ if (c1 != EOF)+ --cur_; // must unget one char+ opcode = *++pc;+ DBGLOG("Take: cap = %zu", cap_);+ continue;+ case 0xFD: // REDO+ cap_ = Const::REDO;+ DBGLOG("Redo");+ cur_ = pos_;+ if (c1 != EOF)+ --cur_; // must unget one char+ opcode = *++pc;+ continue;+ case 0xFC: // TAIL+ {+ Pattern::Lookahead la = Pattern::lookahead_of(opcode);+ DBGLOG("Tail: %u", la);+ if (lap_.size() > la && lap_[la] >= 0)+ cur_ = txt_ - buf_ + static_cast<size_t>(lap_[la]); // mind the (new) gap+ opcode = *++pc;+ continue;+ }+ case 0xFB: // HEAD+ opcode = *++pc;+ continue;+#if !defined(WITH_NO_INDENT)+ case Pattern::META_DED - Pattern::META_MIN:+ DBGLOG("DED? %d", c1);+ if (jump == Pattern::Const::IMAX && back == Pattern::Const::IMAX && bol && dedent())+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case Pattern::META_IND - Pattern::META_MIN:+ DBGLOG("IND? %d", c1);+ if (jump == Pattern::Const::IMAX && back == Pattern::Const::IMAX && bol && indent())+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case Pattern::META_UND - Pattern::META_MIN:+ DBGLOG("UND");+ if (mrk_)+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ mrk_ = false;+ ded_ = 0;+ opcode = *++pc;+ continue;+#endif+ case Pattern::META_EOB - Pattern::META_MIN:+ DBGLOG("EOB? %d", c1);+ if (jump == Pattern::Const::IMAX && c1 == EOF)+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case Pattern::META_BOB - Pattern::META_MIN:+ DBGLOG("BOB? %d", at_bob());+ if (jump == Pattern::Const::IMAX && at_bob())+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case Pattern::META_EOL - Pattern::META_MIN:+ DBGLOG("EOL? %d", c1);+ anc_ = true;+ if (jump == Pattern::Const::IMAX && (c1 == EOF || c1 == '\n' || (c1 == '\r' && peek() == '\n')))+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case Pattern::META_BOL - Pattern::META_MIN:+ DBGLOG("BOL? %d", bol);+ anc_ = true;+ if (jump == Pattern::Const::IMAX && bol)+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case Pattern::META_EWE - Pattern::META_MIN:+ DBGLOG("EWE? %d %d %d", c0, c1, isword(c0) && !isword(c1));+ anc_ = true;+ if (jump == Pattern::Const::IMAX && (isword(c0) || opt_.W) && !isword(c1))+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case Pattern::META_BWE - Pattern::META_MIN:+ DBGLOG("BWE? %d %d %d", c0, c1, !isword(c0) && isword(c1));+ anc_ = true;+ if (jump == Pattern::Const::IMAX && !isword(c0) && isword(c1))+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case Pattern::META_EWB - Pattern::META_MIN:+ DBGLOG("EWB? %d", at_eow());+ anc_ = true;+ if (jump == Pattern::Const::IMAX && isword(got_) &&+ !isword(static_cast<unsigned char>(txt_[len_])))+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case Pattern::META_BWB - Pattern::META_MIN:+ DBGLOG("BWB? %d", at_bow());+ anc_ = true;+ if (jump == Pattern::Const::IMAX && !isword(got_) &&+ (opt_.W || isword(static_cast<unsigned char>(txt_[len_]))))+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case Pattern::META_NWE - Pattern::META_MIN:+ DBGLOG("NWE? %d %d %d", c0, c1, isword(c0) == isword(c1));+ anc_ = true;+ if (jump == Pattern::Const::IMAX && isword(c0) == isword(c1))+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case Pattern::META_NWB - Pattern::META_MIN:+ DBGLOG("NWB? %d %d", at_bow(), at_eow());+ anc_ = true;+ if (jump == Pattern::Const::IMAX &&+ isword(got_) == isword(static_cast<unsigned char>(txt_[len_])))+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case 0xFF: // LONG+ opcode = *++pc;+ continue;+ }+ }+ if (jump == Pattern::Const::IMAX)+ {+ if (back != Pattern::Const::IMAX)+ {+ pc = pat_->opc_ + back;+ opcode = *pc;+ }+ break;+ }+ DBGLOG("Backtrack: pc = %u", jump);+ if (back == Pattern::Const::IMAX)+ back = static_cast<Pattern::Index>(pc - pat_->opc_);+ pc = pat_->opc_ + jump;+ opcode = *pc;+ jump = Pattern::Const::IMAX;+ }+ if (c1 == EOF)+ break;+ }+ else+ {+ if (Pattern::is_opcode_halt(opcode))+ break;+ if (c1 == EOF)+ break;+ c1 = get();+ DBGLOG("Get: c1 = %d", c1);+ if (c1 == EOF)+ break;+ }+ Pattern::Opcode lo = c1 << 24;+ Pattern::Opcode hi = lo | 0x00FFFFFF;+unrolled:+ if (hi < opcode || lo > (opcode << 8))+ {+ opcode = *++pc;+ if (hi < opcode || lo > (opcode << 8))+ {+ opcode = *++pc;+ if (hi < opcode || lo > (opcode << 8))+ {+ opcode = *++pc;+ if (hi < opcode || lo > (opcode << 8))+ {+ opcode = *++pc;+ if (hi < opcode || lo > (opcode << 8))+ {+ opcode = *++pc;+ if (hi < opcode || lo > (opcode << 8))+ {+ opcode = *++pc;+ if (hi < opcode || lo > (opcode << 8))+ {+ opcode = *++pc;+ if (hi < opcode || lo > (opcode << 8))+ {+ opcode = *++pc;+ goto unrolled;+ }+ }+ }+ }+ }+ }+ }+ }+ Pattern::Index jump = Pattern::index_of(opcode);+ if (jump == 0)+ {+ // loop back to start state after only one char matched (one transition) but w/o full match, then optimize+ if (cap_ == 0 && pos_ == cur_ + 1 && method == Const::FIND)+ cur_ = pos_; // set cur_ to move forward from cur_ + 1 with FIND advance()+ }+ else if (jump >= Pattern::Const::LONG)+ {+ if (jump == Pattern::Const::HALT)+ break;+ jump = Pattern::long_index_of(pc[1]);+ }+ pc = pat_->opc_ + jump;+ }+ }+#if !defined(WITH_NO_INDENT)+ if (mrk_ && cap_ != Const::REDO)+ {+ if (col_ > 0 && (tab_.empty() || tab_.back() < col_))+ {+ DBGLOG("Set new stop: tab_[%zu] = %zu", tab_.size(), col_);+ tab_.push_back(col_);+ }+ else if (!tab_.empty() && tab_.back() > col_)+ {+ size_t n;+ for (n = tab_.size() - 1; n > 0; --n)+ if (tab_.at(n - 1) <= col_)+ break;+ ded_ += tab_.size() - n;+ DBGLOG("Dedents: ded = %zu tab_ = %zu", ded_, tab_.size());+ tab_.resize(n);+ // adjust stop when indents are not aligned (Python would give an error though)+ if (n > 0)+ tab_.back() = col_;+ }+ }+ if (ded_ > 0)+ {+ DBGLOG("Dedents: ded = %zu", ded_);+ if (col_ == 0 && bol)+ {+ ded_ += tab_.size();+ tab_.resize(0);+ DBGLOG("Rescan for pending dedents: ded = %zu", ded_);+ pos_ = ind_;+ // avoid looping, match \j exactly+ bol = false;+ goto redo;+ }+ --ded_;+ }+#endif+ if (method == Const::SPLIT)+ {+ DBGLOG("Split: len = %zu cap = %zu cur = %zu pos = %zu end = %zu txt-buf = %zu eob = %d got = %d", len_, cap_, cur_, pos_, end_, txt_-buf_, (int)eof_, got_);+ if (cap_ == 0 || (cur_ == static_cast<size_t>(txt_ - buf_) && !at_bob()))+ {+ if (!hit_end() && (txt_ + len_ < buf_ + end_ || peek() != EOF))+ {+ ++len_;+ DBGLOG("Split continue: len = %zu", len_);+ set_current(++cur_);+ goto find;+ }+ if (got_ != Const::EOB)+ cap_ = Const::EMPTY;+ else+ cap_ = 0;+ set_current(end_);+ got_ = Const::EOB;+ DBGLOG("Split at eof: cap = %zu txt = '%s' len = %zu", cap_, std::string(txt_, len_).c_str(), len_);+ DBGLOG("END Matcher::match()");+ return cap_;+ }+ if (cur_ == 0 && at_bob() && at_end())+ {+ cap_ = Const::EMPTY;+ got_ = Const::EOB;+ }+ else+ {+ set_current(cur_);+ }+ DBGLOG("Split: txt = '%s' len = %zu", std::string(txt_, len_).c_str(), len_);+ DBGLOG("END Matcher::match()");+ return cap_;+ }+ if (cap_ == 0)+ {+ if (method == Const::FIND)+ {+ if (!at_end())+ {+ if (anc_)+ {+ cur_ = txt_ - buf_; // reset current to pattern start when a word boundary was encountered+ anc_ = false;+ }+ if (pos_ > cur_) // if we didn't fail on META alone+ {+ if (+#if defined(COMPILE_AVX512BW)+ simd_advance_avx512bw()+#elif defined(COMPILE_AVX2)+ simd_advance_avx2()+#else+ advance()+#endif+ )+ {+ if (!pat_->one_)+ goto scan;+ txt_ = buf_ + cur_;+ len_ = pat_->len_;+ set_current(cur_ + len_);+ return cap_ = 1;+ }+ }+ txt_ = buf_ + cur_;+ }+ }+ else+ {+ // SCAN and MATCH: no match: backup to begin of unmatched text to report as error+ cur_ = txt_ - buf_;+ }+ }+ len_ = cur_ - (txt_ - buf_);+ if (len_ == 0 && !nul)+ {+ DBGLOG("Empty or no match cur = %zu pos = %zu end = %zu", cur_, pos_, end_);+ pos_ = cur_;+ if (at_end())+ {+ set_current(cur_);+ DBGLOG("Reject empty match at EOF");+ cap_ = 0;+ }+ else if (method == Const::FIND)+ {+ DBGLOG("Reject empty match and continue?");+ // skip one char to keep searching+ set_current(++cur_);+ // allow FIND with "N" to match an empty line, with ^$ etc.+ if (cap_ == 0 || !opt_.N)+ goto scan;+ DBGLOG("Accept empty match");+ }+ else+ {+ set_current(cur_);+ DBGLOG("Reject empty match");+ cap_ = 0;+ }+ }+ else if (len_ == 0 && cur_ == end_)+ {+ DBGLOG("Hit end: got = %d", got_);+ if (cap_ == Const::REDO && !opt_.A)+ cap_ = 0;+ }+ else+ {+ set_current(cur_);+ if (len_ > 0 && cap_ == Const::REDO && !opt_.A)+ {+ DBGLOG("Ignore accept and continue: len = %zu", len_);+ len_ = 0;+ if (method != Const::MATCH)+ goto scan;+ cap_ = 0;+ }+ }+ DBGLOG("Return: cap = %zu txt = '%s' len = %zu pos = %zu got = %d", cap_, std::string(txt_, len_).c_str(), len_, pos_, got_);+ DBGLOG("END match()");+ return cap_;+}++#if defined(COMPILE_AVX512BW)+/// Compile an optimized AVX512BW version defined in matcher_avx512bw.cpp+bool Matcher::simd_advance_avx512bw()+{+#elif defined(COMPILE_AVX2)+/// Compile an optimized AVX2 version defined in matcher_avx2.cpp+bool Matcher::simd_advance_avx2()+{+#else+/// advance input cursor position after mismatch to align input for the next match+bool Matcher::advance()+{+#endif+ size_t loc = cur_ + 1;+ size_t min = pat_->min_;+ if (pat_->len_ == 0)+ {+ if (min == 0)+ return false;+ if (loc + min > end_)+ {+ set_current_match(loc - 1);+ (void)peek_more();+ loc = cur_ + 1;+ if (loc + min > end_)+ return false;+ }+ if (min >= 4)+ {+ const Pattern::Pred *bit = pat_->bit_;+ Pattern::Pred state = ~0;+ Pattern::Pred mask = (1 << (min - 1));+ while (true)+ {+ const char *s = buf_ + loc;+ const char *e = buf_ + end_;+ while (s < e)+ {+ state = (state << 1) | bit[static_cast<uint8_t>(*s)];+ if ((state & mask) == 0)+ break;+ ++s;+ }+ if (s < e)+ {+ s -= min - 1;+ loc = s - buf_;+ if (Pattern::predict_match(pat_->pmh_, s, min))+ {+ set_current(loc);+ return true;+ }+ loc += min;+ }+ else+ {+ loc = s - buf_;+ set_current_match(loc - min);+ (void)peek_more();+ loc = cur_ + min;+ if (loc >= end_)+ return false;+ }+ }+ }+ const Pattern::Pred *pma = pat_->pma_;+ if (min == 3)+ {+ const Pattern::Pred *bit = pat_->bit_;+ Pattern::Pred state = ~0;+ while (true)+ {+ const char *s = buf_ + loc;+ const char *e = buf_ + end_;+ while (s < e)+ {+ state = (state << 1) | bit[static_cast<uint8_t>(*s)];+ if ((state & 4) == 0)+ break;+ ++s;+ }+ if (s < e)+ {+ s -= 2;+ loc = s - buf_;+ if (s + 4 > e || Pattern::predict_match(pma, s) == 0)+ {+ set_current(loc);+ return true;+ }+ loc += 3;+ }+ else+ {+ loc = s - buf_;+ set_current_match(loc - 3);+ (void)peek_more();+ loc = cur_ + 3;+ if (loc >= end_)+ return false;+ }+ }+ }+ if (min == 2)+ {+ const Pattern::Pred *bit = pat_->bit_;+ Pattern::Pred state = ~0;+ while (true)+ {+ const char *s = buf_ + loc;+ const char *e = buf_ + end_;+ while (s < e)+ {+ state = (state << 1) | bit[static_cast<uint8_t>(*s)];+ if ((state & 2) == 0)+ break;+ ++s;+ }+ if (s < e)+ {+ s -= 1;+ loc = s - buf_;+ if (s + 4 > e || Pattern::predict_match(pma, s) == 0)+ {+ set_current(loc);+ return true;+ }+ loc += 2;+ }+ else+ {+ loc = s - buf_;+ set_current_match(loc - 2);+ (void)peek_more();+ loc = cur_ + 2;+ if (loc >= end_)+ return false;+ }+ }+ }+ while (true)+ {+ const char *s = buf_ + loc;+ const char *e = buf_ + end_;+ while (s < e && (pma[static_cast<uint8_t>(*s)] & 0xc0) == 0xc0)+ ++s;+ if (s < e)+ {+ loc = s - buf_;+ if (s + 4 > e)+ {+ set_current(loc);+ return true;+ }+ size_t k = Pattern::predict_match(pma, s);+ if (k == 0)+ {+ set_current(loc);+ return true;+ }+ loc += k;+ }+ else+ {+ loc = s - buf_;+ set_current_match(loc - 1);+ (void)peek_more();+ loc = cur_ + 1;+ if (loc >= end_)+ return false;+ }+ }+ }+ const char *pre = pat_->pre_;+ size_t len = pat_->len_; // actually never more than 255+ if (len == 1)+ {+ while (true)+ {+ const char *s = buf_ + loc;+ const char *e = buf_ + end_;+ s = static_cast<const char*>(std::memchr(s, *pre, e - s));+ if (s != NULL)+ {+ loc = s - buf_;+ set_current(loc);+ return true;+ }+ loc = e - buf_;+ set_current_match(loc - 1);+ (void)peek_more();+ loc = cur_ + 1;+ if (loc + len > end_)+ return false;+ }+ }+ if (bmd_ == 0)+ {+ // Boyer-Moore preprocessing of the given pattern pat of length len, generates bmd_ > 0 and bms_[] shifts.+ // relative frequency table of English letters, source code, and UTF-8 bytes+ static unsigned char freq[256] = "\0\0\0\0\0\0\0\0\0\73\4\0\0\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\73\70\70\1\1\2\2\70\70\70\2\2\70\70\70\2\3\3\3\3\3\3\3\3\3\3\70\70\70\70\70\70\2\35\14\24\26\37\20\17\30\33\11\12\25\22\32\34\15\7\27\31\36\23\13\21\10\16\6\70\1\70\2\70\1\67\46\56\60\72\52\51\62\65\43\44\57\54\64\66\47\41\61\63\71\55\45\53\42\50\40\70\2\70\2\0\47\47\47\47\47\47\47\47\47\47\47\47\47\47\47\47\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\44\44\44\44\44\44\44\44\44\44\44\44\44\44\44\44\0\0\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\46\56\56\56\56\56\56\56\56\56\56\56\56\46\56\56\73\0\0\0\0\0\0\0\0\0\0\0\0\0\0";+ uint8_t n = static_cast<uint8_t>(len); // okay to cast: actually never more than 255+ uint16_t i;+ for (i = 0; i < 256; ++i)+ bms_[i] = n;+ lcp_ = 0;+ lcs_ = n > 1;+ for (i = 0; i < n; ++i)+ {+ uint8_t pch = static_cast<uint8_t>(pre[i]);+ bms_[pch] = static_cast<uint8_t>(n - i - 1);+ if (i > 0)+ {+ if (freq[static_cast<uint8_t>(pre[lcp_])] > freq[pch])+ {+ lcs_ = lcp_;+ lcp_ = i;+ }+ else if (freq[static_cast<uint8_t>(pre[lcs_])] > freq[pch])+ {+ lcs_ = i;+ }+ }+ }+ uint16_t j;+ for (i = n - 1, j = i; j > 0; --j)+ if (pre[j - 1] == pre[i])+ break;+ bmd_ = i - j + 1;+#if defined(HAVE_AVX512BW) || defined(HAVE_AVX2) || defined(HAVE_SSE2) || defined(__SSE2__) || defined(__x86_64__) || _M_IX86_FP == 2+ size_t score = 0;+ for (i = 0; i < n; ++i)+ score += bms_[static_cast<uint8_t>(pre[i])];+ score /= n;+ uint8_t fch = freq[static_cast<uint8_t>(pre[lcp_])];+#if defined(HAVE_AVX512BW) || defined(HAVE_AVX2) || defined(HAVE_SSE2)+ if (!have_HW_SSE2() && !have_HW_AVX2() && !have_HW_AVX512BW())+ {+ // if scoring is high and freq is high, then use our improved Boyer-Moore instead of memchr()+#if defined(__SSE2__) || defined(__x86_64__) || _M_IX86_FP == 2+ // SSE2 is available, expect fast memchr() to use instead of BM+ if (score > 1 && fch > 35 && (score > 3 || fch > 50) && fch + score > 52)+ lcs_ = 0xffff;+#else+ // no SSE2 available, expect slow memchr() and use BM unless score or frequency are too low+ if (fch > 37 || (fch > 8 && score > 0))+ lcs_ = 0xffff;+#endif+ }+#elif defined(__SSE2__) || defined(__x86_64__) || _M_IX86_FP == 2+ // SSE2 is available, expect fast memchr() to use instead of BM+ if (score > 1 && fch > 35 && (score > 3 || fch > 50) && fch + score > 52)+ lcs_ = 0xffff;+#endif+#endif+ }+ while (true)+ {+ if (lcs_ < len)+ {+ const char *s = buf_ + loc + lcp_;+ const char *e = buf_ + end_ + lcp_ - len + 1;+#if defined(COMPILE_AVX512BW)+ // implements AVX512BW string search scheme based on in http://0x80.pl/articles/simd-friendly-karp-rabin.html+ __m512i vlcp = _mm512_set1_epi8(pre[lcp_]);+ __m512i vlcs = _mm512_set1_epi8(pre[lcs_]);+ while (s + 64 <= e)+ {+ __m512i vlcpm = _mm512_loadu_si512(reinterpret_cast<const __m512i*>(s));+ __m512i vlcsm = _mm512_loadu_si512(reinterpret_cast<const __m512i*>(s + lcs_ - lcp_));+ uint64_t mask = _mm512_cmpeq_epi8_mask(vlcp, vlcpm) & _mm512_cmpeq_epi8_mask(vlcs, vlcsm);+ while (mask != 0)+ {+ uint32_t offset = ctzl(mask);+ if (std::memcmp(s - lcp_ + offset, pre, len) == 0)+ {+ loc = s - lcp_ + offset - buf_;+ set_current(loc);+ if (min == 0)+ return true;+ if (min >= 4)+ {+ if (loc + len + min > end_ || Pattern::predict_match(pat_->pmh_, &buf_[loc + len], min))+ return true;+ }+ else+ {+ if (loc + len + 4 > end_ || Pattern::predict_match(pat_->pma_, &buf_[loc + len]) == 0)+ return true;+ }+ }+ mask &= mask - 1;+ }+ s += 64;+ }+#elif defined(COMPILE_AVX2)+ // implements AVX2 string search scheme based on in http://0x80.pl/articles/simd-friendly-karp-rabin.html+ __m256i vlcp = _mm256_set1_epi8(pre[lcp_]);+ __m256i vlcs = _mm256_set1_epi8(pre[lcs_]);+ while (s + 32 <= e)+ {+ __m256i vlcpm = _mm256_loadu_si256(reinterpret_cast<const __m256i*>(s));+ __m256i vlcsm = _mm256_loadu_si256(reinterpret_cast<const __m256i*>(s + lcs_ - lcp_));+ __m256i vlcpeq = _mm256_cmpeq_epi8(vlcp, vlcpm);+ __m256i vlcseq = _mm256_cmpeq_epi8(vlcs, vlcsm);+ uint32_t mask = _mm256_movemask_epi8(_mm256_and_si256(vlcpeq, vlcseq));+ while (mask != 0)+ {+ uint32_t offset = ctz(mask);+ if (std::memcmp(s - lcp_ + offset, pre, len) == 0)+ {+ loc = s - lcp_ + offset - buf_;+ set_current(loc);+ if (min == 0)+ return true;+ if (min >= 4)+ {+ if (loc + len + min > end_ || Pattern::predict_match(pat_->pmh_, &buf_[loc + len], min))+ return true;+ }+ else+ {+ if (loc + len + 4 > end_ || Pattern::predict_match(pat_->pma_, &buf_[loc + len]) == 0)+ return true;+ }+ }+ mask &= mask - 1;+ }+ s += 32;+ }+#elif defined(HAVE_SSE2)+ // implements SSE2 string search scheme based on in http://0x80.pl/articles/simd-friendly-karp-rabin.html+ __m128i vlcp = _mm_set1_epi8(pre[lcp_]);+ __m128i vlcs = _mm_set1_epi8(pre[lcs_]);+ while (s + 16 <= e)+ {+ __m128i vlcpm = _mm_loadu_si128(reinterpret_cast<const __m128i*>(s));+ __m128i vlcsm = _mm_loadu_si128(reinterpret_cast<const __m128i*>(s + lcs_ - lcp_));+ __m128i vlcpeq = _mm_cmpeq_epi8(vlcp, vlcpm);+ __m128i vlcseq = _mm_cmpeq_epi8(vlcs, vlcsm);+ uint32_t mask = _mm_movemask_epi8(_mm_and_si128(vlcpeq, vlcseq));+ while (mask != 0)+ {+ uint32_t offset = ctz(mask);+ if (std::memcmp(s - lcp_ + offset, pre, len) == 0)+ {+ loc = s - lcp_ + offset - buf_;+ set_current(loc);+ if (min == 0)+ return true;+ if (min >= 4)+ {+ if (loc + len + min > end_ || Pattern::predict_match(pat_->pmh_, &buf_[loc + len], min))+ return true;+ }+ else+ {+ if (loc + len + 4 > end_ || Pattern::predict_match(pat_->pma_, &buf_[loc + len]) == 0)+ return true;+ }+ }+ mask &= mask - 1;+ }+ s += 16;+ }+#elif defined(HAVE_NEON)+ // implements NEON/AArch64 string search scheme based on in http://0x80.pl/articles/simd-friendly-karp-rabin.html but 64 bit optimized+ uint8x16_t vlcp = vdupq_n_u8(pre[lcp_]);+ uint8x16_t vlcs = vdupq_n_u8(pre[lcs_]);+ while (s + 16 <= e)+ {+ uint8x16_t vlcpm = vld1q_u8(reinterpret_cast<const uint8_t*>(s));+ uint8x16_t vlcsm = vld1q_u8(reinterpret_cast<const uint8_t*>(s) + lcs_ - lcp_);+ uint8x16_t vlcpeq = vceqq_u8(vlcp, vlcpm);+ uint8x16_t vlcseq = vceqq_u8(vlcs, vlcsm);+ uint8x16_t vmask8 = vandq_u8(vlcpeq, vlcseq);+ uint64x2_t vmask64 = vreinterpretq_u64_u8(vmask8);+ uint64_t mask = vgetq_lane_u64(vmask64, 0);+ if (mask != 0)+ {+ for (int i = 0; i < 8; ++i)+ {+ if ((mask & 0xff) && std::memcmp(s - lcp_ + i, pre, len) == 0)+ {+ loc = s - lcp_ + i - buf_;+ set_current(loc);+ if (min == 0)+ return true;+ if (min >= 4)+ {+ if (loc + len + min > end_ || Pattern::predict_match(pat_->pmh_, &buf_[loc + len], min))+ return true;+ }+ else+ {+ if (loc + len + 4 > end_ || Pattern::predict_match(pat_->pma_, &buf_[loc + len]) == 0)+ return true;+ }+ }+ mask >>= 8;+ }+ }+ mask = vgetq_lane_u64(vmask64, 1);+ if (mask != 0)+ {+ for (int i = 0; i < 8; ++i)+ {+ if ((mask & 0xff) && std::memcmp(s - lcp_ + i + 8, pre, len) == 0)+ {+ loc = s - lcp_ + i + 8 - buf_;+ set_current(loc);+ if (min == 0)+ return true;+ if (min >= 4)+ {+ if (loc + len + min > end_ || Pattern::predict_match(pat_->pmh_, &buf_[loc + len], min))+ return true;+ }+ else+ {+ if (loc + len + 4 > end_ || Pattern::predict_match(pat_->pma_, &buf_[loc + len]) == 0)+ return true;+ }+ }+ mask >>= 8;+ }+ }+ s += 16;+ }+#endif+ while (s < e)+ {+ do+ s = static_cast<const char*>(std::memchr(s, pre[lcp_], e - s));+ while (s != NULL && s[lcs_ - lcp_] != pre[lcs_] && ++s < e);+ if (s == NULL || s >= e)+ {+ s = e;+ break;+ }+ if (len <= 2 || memcmp(s - lcp_, pre, len) == 0)+ {+ loc = s - lcp_ - buf_;+ set_current(loc);+ if (min == 0)+ return true;+ if (min >= 4)+ {+ if (loc + len + min > end_ || Pattern::predict_match(pat_->pmh_, &buf_[loc + len], min))+ return true;+ }+ else+ {+ if (loc + len + 4 > end_ || Pattern::predict_match(pat_->pma_, &buf_[loc + len]) == 0)+ return true;+ }+ }+ ++s;+ }+ loc = s - lcp_ - buf_;+ set_current_match(loc - 1);+ (void)peek_more();+ loc = cur_ + 1;+ if (loc + len > end_)+ return false;+ }+ else+ {+ // implements our improved Boyer-Moore scheme+ const char *s = buf_ + loc + len - 1;+ const char *e = buf_ + end_;+ const char *t = pre + len - 1;+ while (s < e)+ {+ size_t k = 0;+ do+ s += k = bms_[static_cast<uint8_t>(*s)];+ while (k > 0 ? s < e : s[lcp_ - len + 1] != pre[lcp_] && (s += bmd_) < e);+ if (s >= e)+ break;+ const char *p = t - 1;+ const char *q = s - 1;+ while (p >= pre && *p == *q)+ {+ --p;+ --q;+ }+ if (p < pre)+ {+ loc = q - buf_ + 1;+ set_current(loc);+ if (min == 0)+ return true;+ if (min >= 4)+ {+ if (loc + len + min > end_ || Pattern::predict_match(pat_->pmh_, &buf_[loc + len], min))+ return true;+ }+ else+ {+ if (loc + len + 4 > end_ || Pattern::predict_match(pat_->pma_, &buf_[loc + len]) == 0)+ return true;+ }+ }+ if (pre + bmd_ >= p)+ {+ s += bmd_;+ }+ else+ {+ size_t k = bms_[static_cast<uint8_t>(*q)];+ if (p + k > t + bmd_)+ s += k - (t - p);+ else+ s += bmd_;+ }+ }+ s -= len - 1;+ loc = s - buf_;+ set_current_match(loc - 1);+ (void)peek_more();+ loc = cur_ + 1;+ if (loc + len > end_)+ return false;+ }+ }+}++} // namespace reflex++#endif
@@ -0,0 +1,1146 @@+/******************************************************************************\+* Copyright (c) 2016, Robert van Engelen, Genivia Inc. All rights reserved. *+* *+* Redistribution and use in source and binary forms, with or without *+* modification, are permitted provided that the following conditions are met: *+* *+* (1) Redistributions of source code must retain the above copyright notice, *+* this list of conditions and the following disclaimer. *+* *+* (2) Redistributions in binary form must reproduce the above copyright *+* notice, this list of conditions and the following disclaimer in the *+* documentation and/or other materials provided with the distribution. *+* *+* (3) The name of the author may not be used to endorse or promote products *+* derived from this software without specific prior written permission. *+* *+* THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR IMPLIED *+* WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF *+* MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO *+* EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, *+* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, *+* PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; *+* OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, *+* WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR *+* OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF *+* ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. *+\******************************************************************************/++/**+@file matcher.cpp, matcher_avx2.cpp, matcher_avx512bw.cpp (same file)+@brief RE/flex matcher engine+@author Robert van Engelen - engelen@genivia.com+@copyright (c) 2016-2022, Robert van Engelen, Genivia Inc. All rights reserved.+@copyright (c) BSD-3 License - see LICENSE.txt+*/++#if defined(COMPILE_AVX512BW) && !defined(HAVE_AVX512BW)++// appease ranlib "has no symbols"+void matcher_not_compiled_with_avx512bw() { }++#elif defined(COMPILE_AVX2) && !defined(HAVE_AVX2)++// appease ranlib "has no symbols"+void matcher_not_compiled_with_avx2() { }++#else++#include <reflex/matcher.h>++namespace reflex {++/*+ The simd_match_avx512bw() and simd_match_avx2() methods are AVX-optimized+ versions of the match() method. To compile these methods separately with+ the appropriate compilation flags, this file is copied to+ matcher_avx512bw.cpp and matcher_avx2.cpp then compiled with -mavx512bw+ -DCOMPILE_AVX512BW -DHAVE_AVX512BW and with -mavx2 -DCOMPILE_AVX2+ -DHAVE_AVX2, respectively. Likewise, the simd_advance_avx512bw() and+ simd_advance_avx2() methods are optimized versions and separately compiled.+ This approach is preferred over maintaining three separate copies of source+ code files with these methods that only slightly differ. On the other hand,+ combining these versions into one source file means more #if branches.++ If -DHAVE_AVX512BW is not defined, -DCOMPILE_AVX512BW has no effect.+ Likewise, if -DHAVE_AVX2 is not defined, -DCOMPILE_AVX2 has no effect.+*/++#if defined(COMPILE_AVX512BW)+/// Compile an optimized AVX512BW version defined in matcher_avx2.cpp+size_t Matcher::simd_match_avx512bw(Method method)+{+#elif defined(COMPILE_AVX2)+/// Compile an optimized AVX2 version defined in matcher_avx512bw.cpp+size_t Matcher::simd_match_avx2(Method method)+{+#else+/// Returns true if input matched the pattern using method Const::SCAN, Const::FIND, Const::SPLIT, or Const::MATCH.+size_t Matcher::match(Method method)+{+ DBGLOG("BEGIN Matcher::match()");+#if defined(HAVE_AVX512BW) && (!defined(_MSC_VER) || defined(_WIN64))+ if (have_HW_AVX512BW())+ return simd_match_avx512bw(method);+ if (have_HW_AVX2())+ return simd_match_avx2(method);+#elif defined(HAVE_AVX2)+ if (have_HW_AVX2())+ return simd_match_avx2(method);+#endif+#endif+ reset_text();+ len_ = 0; // split text length starts with 0+ anc_ = false; // no word boundary anchor found and applied+scan:+ txt_ = buf_ + cur_;+#if !defined(WITH_NO_INDENT)+ mrk_ = false;+ ind_ = pos_; // ind scans input in buf[] in newline() up to pos - 1+ col_ = 0; // count columns for indent matching+#endif+find:+ int c1 = got_;+ bool bol = at_bol(); // at begin of line?+ if (pat_->fsm_ != NULL)+ fsm_.c1 = c1;+#if !defined(WITH_NO_INDENT)+redo:+#endif+ lap_.resize(0);+ cap_ = 0;+ bool nul = method == Const::MATCH;+ if (pat_->fsm_ != NULL)+ {+ DBGLOG("FSM code %p", pat_->fsm_);+ fsm_.bol = bol;+ fsm_.nul = nul;+ pat_->fsm_(*this);+ nul = fsm_.nul;+ c1 = fsm_.c1;+ }+ else if (pat_->opc_ != NULL)+ {+ const Pattern::Opcode *pc = pat_->opc_;+ while (true)+ {+ Pattern::Opcode opcode = *pc;+ DBGLOG("Fetch: code[%zu] = 0x%08X", pc - pat_->opc_, opcode);+ if (!Pattern::is_opcode_goto(opcode))+ {+ switch (opcode >> 24)+ {+ case 0xFE: // TAKE+ cap_ = Pattern::long_index_of(opcode);+ cur_ = pos_;+ ++pc;+ DBGLOG("Take: cap = %zu", cap_);+ continue;+ case 0xFD: // REDO+ cap_ = Const::REDO;+ DBGLOG("Redo");+ cur_ = pos_;+ ++pc;+ continue;+ case 0xFC: // TAIL+ {+ Pattern::Lookahead la = Pattern::lookahead_of(opcode);+ DBGLOG("Tail: %u", la);+ if (lap_.size() > la && lap_[la] >= 0)+ cur_ = txt_ - buf_ + static_cast<size_t>(lap_[la]); // mind the (new) gap+ ++pc;+ continue;+ }+ case 0xFB: // HEAD+ {+ Pattern::Lookahead la = Pattern::lookahead_of(opcode);+ DBGLOG("Head: lookahead[%u] = %zu", la, pos_ - (txt_ - buf_));+ if (lap_.size() <= la)+ lap_.resize(la + 1, -1);+ lap_[la] = static_cast<int>(pos_ - (txt_ - buf_)); // mind the gap+ ++pc;+ continue;+ }+#if !defined(WITH_NO_INDENT)+ case Pattern::META_DED - Pattern::META_MIN:+ if (ded_ > 0)+ {+ Pattern::Index jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(pc[1]);+ DBGLOG("Dedent ded = %zu", ded_); // unconditional dedent matching \j+ nul = true;+ pc = pat_->opc_ + jump;+ continue;+ }+#endif+ }+ if (c1 == EOF)+ break;+ int c0 = c1;+ c1 = get();+ DBGLOG("Get: c1 = %d", c1);+ Pattern::Index back = Pattern::Const::IMAX; // where to jump back to (backtrack on meta transitions)+ Pattern::Index jump = Pattern::Const::IMAX; // to jump to longest sequence of matching metas+ while (true)+ {+ if ((jump == Pattern::Const::IMAX || back == Pattern::Const::IMAX) && !Pattern::is_opcode_goto(opcode))+ {+ // we no longer have to pass through all if jump and back are set+ switch (opcode >> 24)+ {+ case 0xFE: // TAKE+ cap_ = Pattern::long_index_of(opcode);+ cur_ = pos_;+ if (c1 != EOF)+ --cur_; // must unget one char+ opcode = *++pc;+ DBGLOG("Take: cap = %zu", cap_);+ continue;+ case 0xFD: // REDO+ cap_ = Const::REDO;+ DBGLOG("Redo");+ cur_ = pos_;+ if (c1 != EOF)+ --cur_; // must unget one char+ opcode = *++pc;+ continue;+ case 0xFC: // TAIL+ {+ Pattern::Lookahead la = Pattern::lookahead_of(opcode);+ DBGLOG("Tail: %u", la);+ if (lap_.size() > la && lap_[la] >= 0)+ cur_ = txt_ - buf_ + static_cast<size_t>(lap_[la]); // mind the (new) gap+ opcode = *++pc;+ continue;+ }+ case 0xFB: // HEAD+ opcode = *++pc;+ continue;+#if !defined(WITH_NO_INDENT)+ case Pattern::META_DED - Pattern::META_MIN:+ DBGLOG("DED? %d", c1);+ if (jump == Pattern::Const::IMAX && back == Pattern::Const::IMAX && bol && dedent())+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case Pattern::META_IND - Pattern::META_MIN:+ DBGLOG("IND? %d", c1);+ if (jump == Pattern::Const::IMAX && back == Pattern::Const::IMAX && bol && indent())+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case Pattern::META_UND - Pattern::META_MIN:+ DBGLOG("UND");+ if (mrk_)+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ mrk_ = false;+ ded_ = 0;+ opcode = *++pc;+ continue;+#endif+ case Pattern::META_EOB - Pattern::META_MIN:+ DBGLOG("EOB? %d", c1);+ if (jump == Pattern::Const::IMAX && c1 == EOF)+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case Pattern::META_BOB - Pattern::META_MIN:+ DBGLOG("BOB? %d", at_bob());+ if (jump == Pattern::Const::IMAX && at_bob())+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case Pattern::META_EOL - Pattern::META_MIN:+ DBGLOG("EOL? %d", c1);+ anc_ = true;+ if (jump == Pattern::Const::IMAX && (c1 == EOF || c1 == '\n' || (c1 == '\r' && peek() == '\n')))+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case Pattern::META_BOL - Pattern::META_MIN:+ DBGLOG("BOL? %d", bol);+ anc_ = true;+ if (jump == Pattern::Const::IMAX && bol)+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case Pattern::META_EWE - Pattern::META_MIN:+ DBGLOG("EWE? %d %d %d", c0, c1, isword(c0) && !isword(c1));+ anc_ = true;+ if (jump == Pattern::Const::IMAX && (isword(c0) || opt_.W) && !isword(c1))+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case Pattern::META_BWE - Pattern::META_MIN:+ DBGLOG("BWE? %d %d %d", c0, c1, !isword(c0) && isword(c1));+ anc_ = true;+ if (jump == Pattern::Const::IMAX && !isword(c0) && isword(c1))+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case Pattern::META_EWB - Pattern::META_MIN:+ DBGLOG("EWB? %d", at_eow());+ anc_ = true;+ if (jump == Pattern::Const::IMAX && isword(got_) &&+ !isword(static_cast<unsigned char>(txt_[len_])))+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case Pattern::META_BWB - Pattern::META_MIN:+ DBGLOG("BWB? %d", at_bow());+ anc_ = true;+ if (jump == Pattern::Const::IMAX && !isword(got_) &&+ (opt_.W || isword(static_cast<unsigned char>(txt_[len_]))))+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case Pattern::META_NWE - Pattern::META_MIN:+ DBGLOG("NWE? %d %d %d", c0, c1, isword(c0) == isword(c1));+ anc_ = true;+ if (jump == Pattern::Const::IMAX && isword(c0) == isword(c1))+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case Pattern::META_NWB - Pattern::META_MIN:+ DBGLOG("NWB? %d %d", at_bow(), at_eow());+ anc_ = true;+ if (jump == Pattern::Const::IMAX &&+ isword(got_) == isword(static_cast<unsigned char>(txt_[len_])))+ {+ jump = Pattern::index_of(opcode);+ if (jump == Pattern::Const::LONG)+ jump = Pattern::long_index_of(*++pc);+ }+ opcode = *++pc;+ continue;+ case 0xFF: // LONG+ opcode = *++pc;+ continue;+ }+ }+ if (jump == Pattern::Const::IMAX)+ {+ if (back != Pattern::Const::IMAX)+ {+ pc = pat_->opc_ + back;+ opcode = *pc;+ }+ break;+ }+ DBGLOG("Backtrack: pc = %u", jump);+ if (back == Pattern::Const::IMAX)+ back = static_cast<Pattern::Index>(pc - pat_->opc_);+ pc = pat_->opc_ + jump;+ opcode = *pc;+ jump = Pattern::Const::IMAX;+ }+ if (c1 == EOF)+ break;+ }+ else+ {+ if (Pattern::is_opcode_halt(opcode))+ break;+ if (c1 == EOF)+ break;+ c1 = get();+ DBGLOG("Get: c1 = %d", c1);+ if (c1 == EOF)+ break;+ }+ Pattern::Opcode lo = c1 << 24;+ Pattern::Opcode hi = lo | 0x00FFFFFF;+unrolled:+ if (hi < opcode || lo > (opcode << 8))+ {+ opcode = *++pc;+ if (hi < opcode || lo > (opcode << 8))+ {+ opcode = *++pc;+ if (hi < opcode || lo > (opcode << 8))+ {+ opcode = *++pc;+ if (hi < opcode || lo > (opcode << 8))+ {+ opcode = *++pc;+ if (hi < opcode || lo > (opcode << 8))+ {+ opcode = *++pc;+ if (hi < opcode || lo > (opcode << 8))+ {+ opcode = *++pc;+ if (hi < opcode || lo > (opcode << 8))+ {+ opcode = *++pc;+ if (hi < opcode || lo > (opcode << 8))+ {+ opcode = *++pc;+ goto unrolled;+ }+ }+ }+ }+ }+ }+ }+ }+ Pattern::Index jump = Pattern::index_of(opcode);+ if (jump == 0)+ {+ // loop back to start state after only one char matched (one transition) but w/o full match, then optimize+ if (cap_ == 0 && pos_ == cur_ + 1 && method == Const::FIND)+ cur_ = pos_; // set cur_ to move forward from cur_ + 1 with FIND advance()+ }+ else if (jump >= Pattern::Const::LONG)+ {+ if (jump == Pattern::Const::HALT)+ break;+ jump = Pattern::long_index_of(pc[1]);+ }+ pc = pat_->opc_ + jump;+ }+ }+#if !defined(WITH_NO_INDENT)+ if (mrk_ && cap_ != Const::REDO)+ {+ if (col_ > 0 && (tab_.empty() || tab_.back() < col_))+ {+ DBGLOG("Set new stop: tab_[%zu] = %zu", tab_.size(), col_);+ tab_.push_back(col_);+ }+ else if (!tab_.empty() && tab_.back() > col_)+ {+ size_t n;+ for (n = tab_.size() - 1; n > 0; --n)+ if (tab_.at(n - 1) <= col_)+ break;+ ded_ += tab_.size() - n;+ DBGLOG("Dedents: ded = %zu tab_ = %zu", ded_, tab_.size());+ tab_.resize(n);+ // adjust stop when indents are not aligned (Python would give an error though)+ if (n > 0)+ tab_.back() = col_;+ }+ }+ if (ded_ > 0)+ {+ DBGLOG("Dedents: ded = %zu", ded_);+ if (col_ == 0 && bol)+ {+ ded_ += tab_.size();+ tab_.resize(0);+ DBGLOG("Rescan for pending dedents: ded = %zu", ded_);+ pos_ = ind_;+ // avoid looping, match \j exactly+ bol = false;+ goto redo;+ }+ --ded_;+ }+#endif+ if (method == Const::SPLIT)+ {+ DBGLOG("Split: len = %zu cap = %zu cur = %zu pos = %zu end = %zu txt-buf = %zu eob = %d got = %d", len_, cap_, cur_, pos_, end_, txt_-buf_, (int)eof_, got_);+ if (cap_ == 0 || (cur_ == static_cast<size_t>(txt_ - buf_) && !at_bob()))+ {+ if (!hit_end() && (txt_ + len_ < buf_ + end_ || peek() != EOF))+ {+ ++len_;+ DBGLOG("Split continue: len = %zu", len_);+ set_current(++cur_);+ goto find;+ }+ if (got_ != Const::EOB)+ cap_ = Const::EMPTY;+ else+ cap_ = 0;+ set_current(end_);+ got_ = Const::EOB;+ DBGLOG("Split at eof: cap = %zu txt = '%s' len = %zu", cap_, std::string(txt_, len_).c_str(), len_);+ DBGLOG("END Matcher::match()");+ return cap_;+ }+ if (cur_ == 0 && at_bob() && at_end())+ {+ cap_ = Const::EMPTY;+ got_ = Const::EOB;+ }+ else+ {+ set_current(cur_);+ }+ DBGLOG("Split: txt = '%s' len = %zu", std::string(txt_, len_).c_str(), len_);+ DBGLOG("END Matcher::match()");+ return cap_;+ }+ if (cap_ == 0)+ {+ if (method == Const::FIND)+ {+ if (!at_end())+ {+ if (anc_)+ {+ cur_ = txt_ - buf_; // reset current to pattern start when a word boundary was encountered+ anc_ = false;+ }+ if (pos_ > cur_) // if we didn't fail on META alone+ {+ if (+#if defined(COMPILE_AVX512BW)+ simd_advance_avx512bw()+#elif defined(COMPILE_AVX2)+ simd_advance_avx2()+#else+ advance()+#endif+ )+ {+ if (!pat_->one_)+ goto scan;+ txt_ = buf_ + cur_;+ len_ = pat_->len_;+ set_current(cur_ + len_);+ return cap_ = 1;+ }+ }+ txt_ = buf_ + cur_;+ }+ }+ else+ {+ // SCAN and MATCH: no match: backup to begin of unmatched text to report as error+ cur_ = txt_ - buf_;+ }+ }+ len_ = cur_ - (txt_ - buf_);+ if (len_ == 0 && !nul)+ {+ DBGLOG("Empty or no match cur = %zu pos = %zu end = %zu", cur_, pos_, end_);+ pos_ = cur_;+ if (at_end())+ {+ set_current(cur_);+ DBGLOG("Reject empty match at EOF");+ cap_ = 0;+ }+ else if (method == Const::FIND)+ {+ DBGLOG("Reject empty match and continue?");+ // skip one char to keep searching+ set_current(++cur_);+ // allow FIND with "N" to match an empty line, with ^$ etc.+ if (cap_ == 0 || !opt_.N)+ goto scan;+ DBGLOG("Accept empty match");+ }+ else+ {+ set_current(cur_);+ DBGLOG("Reject empty match");+ cap_ = 0;+ }+ }+ else if (len_ == 0 && cur_ == end_)+ {+ DBGLOG("Hit end: got = %d", got_);+ if (cap_ == Const::REDO && !opt_.A)+ cap_ = 0;+ }+ else+ {+ set_current(cur_);+ if (len_ > 0 && cap_ == Const::REDO && !opt_.A)+ {+ DBGLOG("Ignore accept and continue: len = %zu", len_);+ len_ = 0;+ if (method != Const::MATCH)+ goto scan;+ cap_ = 0;+ }+ }+ DBGLOG("Return: cap = %zu txt = '%s' len = %zu pos = %zu got = %d", cap_, std::string(txt_, len_).c_str(), len_, pos_, got_);+ DBGLOG("END match()");+ return cap_;+}++#if defined(COMPILE_AVX512BW)+/// Compile an optimized AVX512BW version defined in matcher_avx512bw.cpp+bool Matcher::simd_advance_avx512bw()+{+#elif defined(COMPILE_AVX2)+/// Compile an optimized AVX2 version defined in matcher_avx2.cpp+bool Matcher::simd_advance_avx2()+{+#else+/// advance input cursor position after mismatch to align input for the next match+bool Matcher::advance()+{+#endif+ size_t loc = cur_ + 1;+ size_t min = pat_->min_;+ if (pat_->len_ == 0)+ {+ if (min == 0)+ return false;+ if (loc + min > end_)+ {+ set_current_match(loc - 1);+ (void)peek_more();+ loc = cur_ + 1;+ if (loc + min > end_)+ return false;+ }+ if (min >= 4)+ {+ const Pattern::Pred *bit = pat_->bit_;+ Pattern::Pred state = ~0;+ Pattern::Pred mask = (1 << (min - 1));+ while (true)+ {+ const char *s = buf_ + loc;+ const char *e = buf_ + end_;+ while (s < e)+ {+ state = (state << 1) | bit[static_cast<uint8_t>(*s)];+ if ((state & mask) == 0)+ break;+ ++s;+ }+ if (s < e)+ {+ s -= min - 1;+ loc = s - buf_;+ if (Pattern::predict_match(pat_->pmh_, s, min))+ {+ set_current(loc);+ return true;+ }+ loc += min;+ }+ else+ {+ loc = s - buf_;+ set_current_match(loc - min);+ (void)peek_more();+ loc = cur_ + min;+ if (loc >= end_)+ return false;+ }+ }+ }+ const Pattern::Pred *pma = pat_->pma_;+ if (min == 3)+ {+ const Pattern::Pred *bit = pat_->bit_;+ Pattern::Pred state = ~0;+ while (true)+ {+ const char *s = buf_ + loc;+ const char *e = buf_ + end_;+ while (s < e)+ {+ state = (state << 1) | bit[static_cast<uint8_t>(*s)];+ if ((state & 4) == 0)+ break;+ ++s;+ }+ if (s < e)+ {+ s -= 2;+ loc = s - buf_;+ if (s + 4 > e || Pattern::predict_match(pma, s) == 0)+ {+ set_current(loc);+ return true;+ }+ loc += 3;+ }+ else+ {+ loc = s - buf_;+ set_current_match(loc - 3);+ (void)peek_more();+ loc = cur_ + 3;+ if (loc >= end_)+ return false;+ }+ }+ }+ if (min == 2)+ {+ const Pattern::Pred *bit = pat_->bit_;+ Pattern::Pred state = ~0;+ while (true)+ {+ const char *s = buf_ + loc;+ const char *e = buf_ + end_;+ while (s < e)+ {+ state = (state << 1) | bit[static_cast<uint8_t>(*s)];+ if ((state & 2) == 0)+ break;+ ++s;+ }+ if (s < e)+ {+ s -= 1;+ loc = s - buf_;+ if (s + 4 > e || Pattern::predict_match(pma, s) == 0)+ {+ set_current(loc);+ return true;+ }+ loc += 2;+ }+ else+ {+ loc = s - buf_;+ set_current_match(loc - 2);+ (void)peek_more();+ loc = cur_ + 2;+ if (loc >= end_)+ return false;+ }+ }+ }+ while (true)+ {+ const char *s = buf_ + loc;+ const char *e = buf_ + end_;+ while (s < e && (pma[static_cast<uint8_t>(*s)] & 0xc0) == 0xc0)+ ++s;+ if (s < e)+ {+ loc = s - buf_;+ if (s + 4 > e)+ {+ set_current(loc);+ return true;+ }+ size_t k = Pattern::predict_match(pma, s);+ if (k == 0)+ {+ set_current(loc);+ return true;+ }+ loc += k;+ }+ else+ {+ loc = s - buf_;+ set_current_match(loc - 1);+ (void)peek_more();+ loc = cur_ + 1;+ if (loc >= end_)+ return false;+ }+ }+ }+ const char *pre = pat_->pre_;+ size_t len = pat_->len_; // actually never more than 255+ if (len == 1)+ {+ while (true)+ {+ const char *s = buf_ + loc;+ const char *e = buf_ + end_;+ s = static_cast<const char*>(std::memchr(s, *pre, e - s));+ if (s != NULL)+ {+ loc = s - buf_;+ set_current(loc);+ return true;+ }+ loc = e - buf_;+ set_current_match(loc - 1);+ (void)peek_more();+ loc = cur_ + 1;+ if (loc + len > end_)+ return false;+ }+ }+ if (bmd_ == 0)+ {+ // Boyer-Moore preprocessing of the given pattern pat of length len, generates bmd_ > 0 and bms_[] shifts.+ // relative frequency table of English letters, source code, and UTF-8 bytes+ static unsigned char freq[256] = "\0\0\0\0\0\0\0\0\0\73\4\0\0\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\73\70\70\1\1\2\2\70\70\70\2\2\70\70\70\2\3\3\3\3\3\3\3\3\3\3\70\70\70\70\70\70\2\35\14\24\26\37\20\17\30\33\11\12\25\22\32\34\15\7\27\31\36\23\13\21\10\16\6\70\1\70\2\70\1\67\46\56\60\72\52\51\62\65\43\44\57\54\64\66\47\41\61\63\71\55\45\53\42\50\40\70\2\70\2\0\47\47\47\47\47\47\47\47\47\47\47\47\47\47\47\47\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\45\44\44\44\44\44\44\44\44\44\44\44\44\44\44\44\44\0\0\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\5\46\56\56\56\56\56\56\56\56\56\56\56\56\46\56\56\73\0\0\0\0\0\0\0\0\0\0\0\0\0\0";+ uint8_t n = static_cast<uint8_t>(len); // okay to cast: actually never more than 255+ uint16_t i;+ for (i = 0; i < 256; ++i)+ bms_[i] = n;+ lcp_ = 0;+ lcs_ = n > 1;+ for (i = 0; i < n; ++i)+ {+ uint8_t pch = static_cast<uint8_t>(pre[i]);+ bms_[pch] = static_cast<uint8_t>(n - i - 1);+ if (i > 0)+ {+ if (freq[static_cast<uint8_t>(pre[lcp_])] > freq[pch])+ {+ lcs_ = lcp_;+ lcp_ = i;+ }+ else if (freq[static_cast<uint8_t>(pre[lcs_])] > freq[pch])+ {+ lcs_ = i;+ }+ }+ }+ uint16_t j;+ for (i = n - 1, j = i; j > 0; --j)+ if (pre[j - 1] == pre[i])+ break;+ bmd_ = i - j + 1;+#if defined(HAVE_AVX512BW) || defined(HAVE_AVX2) || defined(HAVE_SSE2) || defined(__SSE2__) || defined(__x86_64__) || _M_IX86_FP == 2+ size_t score = 0;+ for (i = 0; i < n; ++i)+ score += bms_[static_cast<uint8_t>(pre[i])];+ score /= n;+ uint8_t fch = freq[static_cast<uint8_t>(pre[lcp_])];+#if defined(HAVE_AVX512BW) || defined(HAVE_AVX2) || defined(HAVE_SSE2)+ if (!have_HW_SSE2() && !have_HW_AVX2() && !have_HW_AVX512BW())+ {+ // if scoring is high and freq is high, then use our improved Boyer-Moore instead of memchr()+#if defined(__SSE2__) || defined(__x86_64__) || _M_IX86_FP == 2+ // SSE2 is available, expect fast memchr() to use instead of BM+ if (score > 1 && fch > 35 && (score > 3 || fch > 50) && fch + score > 52)+ lcs_ = 0xffff;+#else+ // no SSE2 available, expect slow memchr() and use BM unless score or frequency are too low+ if (fch > 37 || (fch > 8 && score > 0))+ lcs_ = 0xffff;+#endif+ }+#elif defined(__SSE2__) || defined(__x86_64__) || _M_IX86_FP == 2+ // SSE2 is available, expect fast memchr() to use instead of BM+ if (score > 1 && fch > 35 && (score > 3 || fch > 50) && fch + score > 52)+ lcs_ = 0xffff;+#endif+#endif+ }+ while (true)+ {+ if (lcs_ < len)+ {+ const char *s = buf_ + loc + lcp_;+ const char *e = buf_ + end_ + lcp_ - len + 1;+#if defined(COMPILE_AVX512BW)+ // implements AVX512BW string search scheme based on in http://0x80.pl/articles/simd-friendly-karp-rabin.html+ __m512i vlcp = _mm512_set1_epi8(pre[lcp_]);+ __m512i vlcs = _mm512_set1_epi8(pre[lcs_]);+ while (s + 64 <= e)+ {+ __m512i vlcpm = _mm512_loadu_si512(reinterpret_cast<const __m512i*>(s));+ __m512i vlcsm = _mm512_loadu_si512(reinterpret_cast<const __m512i*>(s + lcs_ - lcp_));+ uint64_t mask = _mm512_cmpeq_epi8_mask(vlcp, vlcpm) & _mm512_cmpeq_epi8_mask(vlcs, vlcsm);+ while (mask != 0)+ {+ uint32_t offset = ctzl(mask);+ if (std::memcmp(s - lcp_ + offset, pre, len) == 0)+ {+ loc = s - lcp_ + offset - buf_;+ set_current(loc);+ if (min == 0)+ return true;+ if (min >= 4)+ {+ if (loc + len + min > end_ || Pattern::predict_match(pat_->pmh_, &buf_[loc + len], min))+ return true;+ }+ else+ {+ if (loc + len + 4 > end_ || Pattern::predict_match(pat_->pma_, &buf_[loc + len]) == 0)+ return true;+ }+ }+ mask &= mask - 1;+ }+ s += 64;+ }+#elif defined(COMPILE_AVX2)+ // implements AVX2 string search scheme based on in http://0x80.pl/articles/simd-friendly-karp-rabin.html+ __m256i vlcp = _mm256_set1_epi8(pre[lcp_]);+ __m256i vlcs = _mm256_set1_epi8(pre[lcs_]);+ while (s + 32 <= e)+ {+ __m256i vlcpm = _mm256_loadu_si256(reinterpret_cast<const __m256i*>(s));+ __m256i vlcsm = _mm256_loadu_si256(reinterpret_cast<const __m256i*>(s + lcs_ - lcp_));+ __m256i vlcpeq = _mm256_cmpeq_epi8(vlcp, vlcpm);+ __m256i vlcseq = _mm256_cmpeq_epi8(vlcs, vlcsm);+ uint32_t mask = _mm256_movemask_epi8(_mm256_and_si256(vlcpeq, vlcseq));+ while (mask != 0)+ {+ uint32_t offset = ctz(mask);+ if (std::memcmp(s - lcp_ + offset, pre, len) == 0)+ {+ loc = s - lcp_ + offset - buf_;+ set_current(loc);+ if (min == 0)+ return true;+ if (min >= 4)+ {+ if (loc + len + min > end_ || Pattern::predict_match(pat_->pmh_, &buf_[loc + len], min))+ return true;+ }+ else+ {+ if (loc + len + 4 > end_ || Pattern::predict_match(pat_->pma_, &buf_[loc + len]) == 0)+ return true;+ }+ }+ mask &= mask - 1;+ }+ s += 32;+ }+#elif defined(HAVE_SSE2)+ // implements SSE2 string search scheme based on in http://0x80.pl/articles/simd-friendly-karp-rabin.html+ __m128i vlcp = _mm_set1_epi8(pre[lcp_]);+ __m128i vlcs = _mm_set1_epi8(pre[lcs_]);+ while (s + 16 <= e)+ {+ __m128i vlcpm = _mm_loadu_si128(reinterpret_cast<const __m128i*>(s));+ __m128i vlcsm = _mm_loadu_si128(reinterpret_cast<const __m128i*>(s + lcs_ - lcp_));+ __m128i vlcpeq = _mm_cmpeq_epi8(vlcp, vlcpm);+ __m128i vlcseq = _mm_cmpeq_epi8(vlcs, vlcsm);+ uint32_t mask = _mm_movemask_epi8(_mm_and_si128(vlcpeq, vlcseq));+ while (mask != 0)+ {+ uint32_t offset = ctz(mask);+ if (std::memcmp(s - lcp_ + offset, pre, len) == 0)+ {+ loc = s - lcp_ + offset - buf_;+ set_current(loc);+ if (min == 0)+ return true;+ if (min >= 4)+ {+ if (loc + len + min > end_ || Pattern::predict_match(pat_->pmh_, &buf_[loc + len], min))+ return true;+ }+ else+ {+ if (loc + len + 4 > end_ || Pattern::predict_match(pat_->pma_, &buf_[loc + len]) == 0)+ return true;+ }+ }+ mask &= mask - 1;+ }+ s += 16;+ }+#elif defined(HAVE_NEON)+ // implements NEON/AArch64 string search scheme based on in http://0x80.pl/articles/simd-friendly-karp-rabin.html but 64 bit optimized+ uint8x16_t vlcp = vdupq_n_u8(pre[lcp_]);+ uint8x16_t vlcs = vdupq_n_u8(pre[lcs_]);+ while (s + 16 <= e)+ {+ uint8x16_t vlcpm = vld1q_u8(reinterpret_cast<const uint8_t*>(s));+ uint8x16_t vlcsm = vld1q_u8(reinterpret_cast<const uint8_t*>(s) + lcs_ - lcp_);+ uint8x16_t vlcpeq = vceqq_u8(vlcp, vlcpm);+ uint8x16_t vlcseq = vceqq_u8(vlcs, vlcsm);+ uint8x16_t vmask8 = vandq_u8(vlcpeq, vlcseq);+ uint64x2_t vmask64 = vreinterpretq_u64_u8(vmask8);+ uint64_t mask = vgetq_lane_u64(vmask64, 0);+ if (mask != 0)+ {+ for (int i = 0; i < 8; ++i)+ {+ if ((mask & 0xff) && std::memcmp(s - lcp_ + i, pre, len) == 0)+ {+ loc = s - lcp_ + i - buf_;+ set_current(loc);+ if (min == 0)+ return true;+ if (min >= 4)+ {+ if (loc + len + min > end_ || Pattern::predict_match(pat_->pmh_, &buf_[loc + len], min))+ return true;+ }+ else+ {+ if (loc + len + 4 > end_ || Pattern::predict_match(pat_->pma_, &buf_[loc + len]) == 0)+ return true;+ }+ }+ mask >>= 8;+ }+ }+ mask = vgetq_lane_u64(vmask64, 1);+ if (mask != 0)+ {+ for (int i = 0; i < 8; ++i)+ {+ if ((mask & 0xff) && std::memcmp(s - lcp_ + i + 8, pre, len) == 0)+ {+ loc = s - lcp_ + i + 8 - buf_;+ set_current(loc);+ if (min == 0)+ return true;+ if (min >= 4)+ {+ if (loc + len + min > end_ || Pattern::predict_match(pat_->pmh_, &buf_[loc + len], min))+ return true;+ }+ else+ {+ if (loc + len + 4 > end_ || Pattern::predict_match(pat_->pma_, &buf_[loc + len]) == 0)+ return true;+ }+ }+ mask >>= 8;+ }+ }+ s += 16;+ }+#endif+ while (s < e)+ {+ do+ s = static_cast<const char*>(std::memchr(s, pre[lcp_], e - s));+ while (s != NULL && s[lcs_ - lcp_] != pre[lcs_] && ++s < e);+ if (s == NULL || s >= e)+ {+ s = e;+ break;+ }+ if (len <= 2 || memcmp(s - lcp_, pre, len) == 0)+ {+ loc = s - lcp_ - buf_;+ set_current(loc);+ if (min == 0)+ return true;+ if (min >= 4)+ {+ if (loc + len + min > end_ || Pattern::predict_match(pat_->pmh_, &buf_[loc + len], min))+ return true;+ }+ else+ {+ if (loc + len + 4 > end_ || Pattern::predict_match(pat_->pma_, &buf_[loc + len]) == 0)+ return true;+ }+ }+ ++s;+ }+ loc = s - lcp_ - buf_;+ set_current_match(loc - 1);+ (void)peek_more();+ loc = cur_ + 1;+ if (loc + len > end_)+ return false;+ }+ else+ {+ // implements our improved Boyer-Moore scheme+ const char *s = buf_ + loc + len - 1;+ const char *e = buf_ + end_;+ const char *t = pre + len - 1;+ while (s < e)+ {+ size_t k = 0;+ do+ s += k = bms_[static_cast<uint8_t>(*s)];+ while (k > 0 ? s < e : s[lcp_ - len + 1] != pre[lcp_] && (s += bmd_) < e);+ if (s >= e)+ break;+ const char *p = t - 1;+ const char *q = s - 1;+ while (p >= pre && *p == *q)+ {+ --p;+ --q;+ }+ if (p < pre)+ {+ loc = q - buf_ + 1;+ set_current(loc);+ if (min == 0)+ return true;+ if (min >= 4)+ {+ if (loc + len + min > end_ || Pattern::predict_match(pat_->pmh_, &buf_[loc + len], min))+ return true;+ }+ else+ {+ if (loc + len + 4 > end_ || Pattern::predict_match(pat_->pma_, &buf_[loc + len]) == 0)+ return true;+ }+ }+ if (pre + bmd_ >= p)+ {+ s += bmd_;+ }+ else+ {+ size_t k = bms_[static_cast<uint8_t>(*q)];+ if (p + k > t + bmd_)+ s += k - (t - p);+ else+ s += bmd_;+ }+ }+ s -= len - 1;+ loc = s - buf_;+ set_current_match(loc - 1);+ (void)peek_more();+ loc = cur_ + 1;+ if (loc + len > end_)+ return false;+ }+ }+}++} // namespace reflex++#endif
@@ -30,11 +30,11 @@ @file simd_avx2.cpp @brief RE/flex SIMD intrinsics compiled with -mavx2 @author Robert van Engelen - engelen@genivia.com-@copyright (c) 2016-2021, Robert van Engelen, Genivia Inc. All rights reserved.+@copyright (c) 2016-2022, Robert van Engelen, Genivia Inc. All rights reserved. @copyright (c) BSD-3 License - see LICENSE.txt */ -#include <reflex/matcher.h>+#include <reflex/absmatcher.h> namespace reflex { @@ -60,49 +60,4 @@ size_t simd_nlcount_avx2(const char*& b, const char *e) return n; } -// string search scheme based on in http://0x80.pl/articles/simd-friendly-karp-rabin.html-bool Matcher::simd_advance_avx2(const char*& b, const char *e, size_t &loc, size_t min, const char *pre, size_t len)-{- const char *s = b;-#if defined(HAVE_AVX2)- __m256i vlcp = _mm256_set1_epi8(pre[lcp_]);- __m256i vlcs = _mm256_set1_epi8(pre[lcs_]);- while (s + 32 <= e)- {- __m256i vlcpm = _mm256_loadu_si256(reinterpret_cast<const __m256i*>(s));- __m256i vlcsm = _mm256_loadu_si256(reinterpret_cast<const __m256i*>(s + lcs_ - lcp_));- __m256i vlcpeq = _mm256_cmpeq_epi8(vlcp, vlcpm);- __m256i vlcseq = _mm256_cmpeq_epi8(vlcs, vlcsm);- uint32_t mask = _mm256_movemask_epi8(_mm256_and_si256(vlcpeq, vlcseq));- while (mask != 0)- {- uint32_t offset = ctz(mask);- if (std::memcmp(s - lcp_ + offset, pre, len) == 0)- {- loc = s - lcp_ + offset - buf_;- set_current(loc);- if (min == 0)- return true;- if (min >= 4)- {- if (loc + len + min > end_ || Pattern::predict_match(pat_->pmh_, &buf_[loc + len], min))- return true;- }- else- {- if (loc + len + 4 > end_ || Pattern::predict_match(pat_->pma_, &buf_[loc + len]) == 0)- return true;- }- }- mask &= mask - 1;- }- s += 32;- }-#else- (void)e, (void)loc, (void)min, (void)pre, (void)len;-#endif- b = s;- return false;-}- } // namespace reflex
@@ -30,11 +30,11 @@ @file simd_avx512bw.cpp @brief RE/flex SIMD intrinsics compiled with -mavx512bw @author Robert van Engelen - engelen@genivia.com-@copyright (c) 2016-2021, Robert van Engelen, Genivia Inc. All rights reserved.+@copyright (c) 2016-2022, Robert van Engelen, Genivia Inc. All rights reserved. @copyright (c) BSD-3 License - see LICENSE.txt */ -#include <reflex/matcher.h>+#include <reflex/absmatcher.h> namespace reflex { @@ -59,47 +59,4 @@ size_t simd_nlcount_avx512bw(const char*& b, const char *e) return n; } -// string search scheme based on in http://0x80.pl/articles/simd-friendly-karp-rabin.html-bool Matcher::simd_advance_avx512bw(const char*& b, const char *e, size_t &loc, size_t min, const char *pre, size_t len)-{- const char *s = b;-#if defined(HAVE_AVX512BW) && (!defined(_MSC_VER) || defined(_WIN64))- __m512i vlcp = _mm512_set1_epi8(pre[lcp_]);- __m512i vlcs = _mm512_set1_epi8(pre[lcs_]);- while (s + 64 <= e)- {- __m512i vlcpm = _mm512_loadu_si512(reinterpret_cast<const __m512i*>(s));- __m512i vlcsm = _mm512_loadu_si512(reinterpret_cast<const __m512i*>(s + lcs_ - lcp_));- uint64_t mask = _mm512_cmpeq_epi8_mask(vlcp, vlcpm) & _mm512_cmpeq_epi8_mask(vlcs, vlcsm);- while (mask != 0)- {- uint32_t offset = ctzl(mask);- if (std::memcmp(s - lcp_ + offset, pre, len) == 0)- {- loc = s - lcp_ + offset - buf_;- set_current(loc);- if (min == 0)- return true;- if (min >= 4)- {- if (loc + len + min > end_ || Pattern::predict_match(pat_->pmh_, &buf_[loc + len], min))- return true;- }- else- {- if (loc + len + 4 > end_ || Pattern::predict_match(pat_->pma_, &buf_[loc + len]) == 0)- return true;- }- }- mask &= mask - 1;- }- s += 64;- }-#else- (void)e, (void)loc, (void)min, (void)pre, (void)len;-#endif- b = s;- return false;-}- } // namespace reflex
@@ -59,7 +59,10 @@ Option \fB-f FILE\fR matches patterns specified in \fBFILE\fR. If \fBFILE\fR is large and defines complex regular expression patterns, then option \fB-P\fR (Perl matching) may improve performance (this omits POSIX DFA construction.) .PP-For help, \fB--help \fIWHAT\fR displays help on options related to \fIWHAT\fR.+\fBugrep --help \fIWHAT\fR displays help on options related to \fIWHAT\fR;+\fB--help format\fR displays help on \fB--format\fR and \fB--replace\fR+formatting; \fB--help regex\fR displays help on regular expression syntax and+conventions; \fB--help globs\fR displays help on glob patterns. .PP The following options are available: END
@@ -1,4 +1,4 @@-.TH UGREP "1" "January 05, 2022" "ugrep 3.5.0" "User Commands"+.TH UGREP "1" "January 12, 2022" "ugrep 3.6.0" "User Commands" .SH NAME \fBugrep\fR, \fBug\fR -- file pattern searcher .SH SYNOPSIS@@ -44,14 +44,17 @@ Option \fB-f FILE\fR matches patterns specified in \fBFILE\fR. If \fBFILE\fR is large and defines complex regular expression patterns, then option \fB-P\fR (Perl matching) may improve performance (this omits POSIX DFA construction.) .PP-For help, \fB--help \fIWHAT\fR displays help on options related to \fIWHAT\fR.+\fBugrep --help \fIWHAT\fR displays help on options related to \fIWHAT\fR;+\fB--help format\fR displays help on \fB--format\fR and \fB--replace\fR+formatting; \fB--help regex\fR displays help on regular expression syntax and+conventions; \fB--help globs\fR displays help on glob patterns. .PP The following options are available: .TP \fB\-A\fR \fINUM\fR, \fB\-\-after\-context\fR=\fINUM\fR Print NUM lines of trailing context after matching lines. Places a \fB\-\-group\-separator\fR between contiguous groups of matches. See also-options \fB\-B\fR, \fB\-C\fR, and \fB\-y\fR.+options \fB\-B\fR, \fB\-C\fR and \fB\-y\fR. .TP \fB\-a\fR, \fB\-\-text\fR Process a binary file as if it were text. This is equivalent to@@ -67,26 +70,26 @@ pattern. For example, \fB\-e\fR \fIA\fR \fB\-e\fR \fIB\fR \fB\-\-and\fR \fB\-e\ (`A' or `B') and (`C' or `D'). Note that multiple \fB\-e\fR \fIPATTERN\fR are alternations that bind more tightly together than \fB\-\-and\fR. Option \fB\-\-stats\fR displays the search patterns applied. See also options-\fB\-\-not\fR, \fB\-\-andnot\fR, \fB\-\-bool\fR, \fB\-\-files\fR, and \fB\-\-lines\fR.+\fB\-\-not\fR, \fB\-\-andnot\fR, \fB\-\-bool\fR, \fB\-\-files\fR and \fB\-\-lines\fR. .TP \fB\-\-andnot\fR [[\fB\-e\fR] PATTERN] ...-Combines \fB\-\-and\fR \fB\-\-not\fR. See also options \fB\-\-and\fR, \fB\-\-not\fR, and \fB\-\-bool\fR.+Combines \fB\-\-and\fR \fB\-\-not\fR. See also options \fB\-\-and\fR, \fB\-\-not\fR and \fB\-\-bool\fR. .TP \fB\-B\fR \fINUM\fR, \fB\-\-before\-context\fR=\fINUM\fR Print NUM lines of leading context before matching lines. Places a \fB\-\-group\-separator\fR between contiguous groups of matches. See also-options \fB\-A\fR, \fB\-C\fR, and \fB\-y\fR.+options \fB\-A\fR, \fB\-C\fR and \fB\-y\fR. .TP \fB\-b\fR, \fB\-\-byte\-offset\fR The offset in bytes of a matched line is displayed in front of the respective matched line. If \fB\-u\fR is specified, displays the offset for each pattern matched on the same line. Byte offsets are exact-for ASCII, UTF\-8, and raw binary input. Otherwise, the byte offset+for ASCII, UTF\-8 and raw binary input. Otherwise, the byte offset in the UTF\-8 normalized input is displayed. .TP \fB\-\-binary\-files\fR=\fITYPE\fR Controls searching and reporting pattern matches in binary files.-TYPE can be `binary', `without\-match`, `text`, `hex`, and+TYPE can be `binary', `without\-match`, `text`, `hex` and `with\-hex'. The default is `binary' to search binary files and to report a match without displaying the match. `without\-match' ignores binary matches. `text' treats all binary files as text,@@ -95,13 +98,13 @@ problematic consequences if the terminal driver interprets some of it as commands. `hex' reports all matches in hexadecimal. `with\-hex' only reports binary matches in hexadecimal, leaving text matches alone. A match is considered binary when matching a zero-byte or invalid UTF. Short options are \fB\-a\fR, \fB\-I\fR, \fB\-U\fR, \fB\-W\fR, and \fB\-X\fR.+byte or invalid UTF. Short options are \fB\-a\fR, \fB\-I\fR, \fB\-U\fR, \fB\-W\fR and \fB\-X\fR. .TP \fB\-\-bool\fR, \fB\-%\fR Specifies Boolean query patterns. A Boolean query pattern is composed of `AND', `OR', `NOT' operators and grouping with `(' `)'. Spacing between subpatterns is the same as `AND', `|' is the same-as `OR', and a `\-' is the same as `NOT'. The `OR' operator binds+as `OR' and a `\-' is the same as `NOT'. The `OR' operator binds more tightly than `AND'. For example, \fB\-\-bool\fR 'A|B C|D' matches lines with (`A' or `B') and (`C' or `D'), \fB\-\-bool\fR 'A \fB\-B\fR' matches lines with `A' and not `B'. Operators `AND', `OR', `NOT' require@@ -113,11 +116,11 @@ lines with `A' and also either `AND' or `OR'. Parenthesis are used for grouping. For example, \fB\-\-bool\fR '(A B)|C' matches lines with `A' and `B', or lines with `C'. Note that all subpatterns in a Boolean query pattern are regular expressions, unless option \fB\-F\fR is used.-Options \fB\-E\fR, \fB\-F\fR, \fB\-G\fR, \fB\-P\fR, and \fB\-Z\fR can be combined with \fB\-\-bool\fR to match+Options \fB\-E\fR, \fB\-F\fR, \fB\-G\fR, \fB\-P\fR and \fB\-Z\fR can be combined with \fB\-\-bool\fR to match subpatterns as strings or regular expressions (\fB\-E\fR is the default.) This option does not apply to \fB\-f\fR \fIFILE\fR patterns. Option \fB\-\-stats\fR displays the search patterns applied. See also options \fB\-\-and\fR,-\fB\-\-andnot\fR, \fB\-\-not\fR, \fB\-\-files\fR, and \fB\-\-lines\fR.+\fB\-\-andnot\fR, \fB\-\-not\fR, \fB\-\-files\fR and \fB\-\-lines\fR. .TP \fB\-\-break\fR Adds a line break between results from different files.@@ -125,7 +128,7 @@ Adds a line break between results from different files. \fB\-C\fR \fINUM\fR, \fB\-\-context\fR=\fINUM\fR Print NUM lines of leading and trailing context surrounding each match. Places a \fB\-\-group\-separator\fR between contiguous groups of-matches. See also options \fB\-A\fR, \fB\-B\fR, and \fB\-y\fR.+matches. See also options \fB\-A\fR, \fB\-B\fR and \fB\-y\fR. .TP \fB\-c\fR, \fB\-\-count\fR Only a count of selected lines is written to standard output.@@ -224,7 +227,7 @@ The encoding format of the input, where ENCODING can be: .TP \fB\-\-exclude\fR=\fIGLOB\fR Skip files whose name matches GLOB using wildcard matching, same as-\fB\-g\fR ^GLOB. GLOB can use **, *, ?, and [...] as wildcards, and \\ to+\fB\-g\fR ^GLOB. GLOB can use **, *, ?, and [...] as wildcards and \\ to quote a wildcard or backslash character literally. When GLOB contains a `/', full pathnames are matched. Otherwise basenames are matched. When GLOB ends with a `/', directories are excluded@@ -236,7 +239,7 @@ be repeated. \fB\-\-exclude\-dir\fR=\fIGLOB\fR Exclude directories whose name matches GLOB from recursive searches, same as \fB\-g\fR ^GLOB/. GLOB can use **, *, ?, and [...] as-wildcards, and \\ to quote a wildcard or backslash character+wildcards and \\ to quote a wildcard or backslash character literally. When GLOB contains a `/', full pathnames are matched. Otherwise basenames are matched. Note that \fB\-\-exclude\-dir\fR patterns take priority over \fB\-\-include\-dir\fR patterns. GLOB should be quoted@@ -245,7 +248,7 @@ to prevent shell globbing. This option may be repeated. \fB\-\-exclude\-from\fR=\fIFILE\fR Read the globs from FILE and skip files and directories whose name matches one or more globs. A glob can use **, *, ?, and [...] as-wildcards, and \\ to quote a wildcard or backslash character+wildcards and \\ to quote a wildcard or backslash character literally. When a glob contains a `/', full pathnames are matched. Otherwise basenames are matched. When a glob ends with a `/', directories are excluded as if \fB\-\-exclude\-dir\fR is specified.@@ -299,8 +302,11 @@ is invoked. This option may be repeated. \fB\-\-format\fR=\fIFORMAT\fR Output FORMAT\-formatted matches. For example \fB\-\-format\fR='%f:%n:%O%~' outputs matching lines `%O' with filename `%f` and line number `%n'-followed by a newline `%~'. Context options \fB\-A\fR, \fB\-B\fR, \fB\-C\fR, and \fB\-y\fR are-ignored. See `man ugrep' section FORMAT.+followed by a newline `%~'. When option \fB\-P\fR is used, FORMAT may+include `%1' to `%9', `%[\fINUM\fR]#' and `%[\fINAME\fR]#' to output group+captures. A `%%' outputs `%'. See `ugrep \fB\-\-help\fR format' and `man+ugrep' section FORMAT for details. Context options \fB\-A\fR, \fB\-B\fR, \fB\-C\fR and+\fB\-y\fR are ignored. .TP \fB\-\-free\-space\fR Spacing (blanks and tabs) in regular expressions are ignored.@@ -322,7 +328,7 @@ be combined with options \fB\-M\fR, \fB\-O\fR and \fB\-t\fR to expand the recurs search. .TP \fB\-\-group\-separator\fR[=\fISEP\fR]-Use SEP as a group separator for context options \fB\-A\fR, \fB\-B\fR, and \fB\-C\fR.+Use SEP as a group separator for context options \fB\-A\fR, \fB\-B\fR and \fB\-C\fR. The default is a double hyphen (`\fB\-\-\fR'). .TP \fB\-H\fR, \fB\-\-with\-filename\fR@@ -339,6 +345,9 @@ results from different files. .TP \fB\-\-help\fR [\fIWHAT\fR], \fB\-?\fR [\fIWHAT\fR] Display a help message, specifically on WHAT when specified.+In addition, `\fB\-\-help\fR format' displays an overview of FORMAT fields,+`\fB\-\-help\fR regex' displays an overview of regular expressions and+`\fB\-\-help\fR globs' displays an overview of glob syntax and conventions. .TP \fB\-\-hexdump\fR=[1\-8][a][b][c][h] Output matches in 1 to 8 columns of 8 hexadecimal octets. The@@ -378,19 +387,19 @@ arguments are never ignored. This option may be repeated. .TP \fB\-\-include\fR=\fIGLOB\fR Search only files whose name matches GLOB using wildcard matching,-same as \fB\-g\fR \fIGLOB\fR. GLOB can use **, *, ?, and [...] as wildcards,-and \\ to quote a wildcard or backslash character literally. When-GLOB contains a `/', full pathnames are matched. Otherwise-basenames are matched. When GLOB ends with a `/', directories are-included as if \fB\-\-include\-dir\fR is specified. Otherwise files are-included. Note that \fB\-\-exclude\fR patterns take priority over-\fB\-\-include\fR patterns. GLOB should be quoted to prevent shell-globbing. This option may be repeated.+same as \fB\-g\fR \fIGLOB\fR. GLOB can use **, *, ?, and [...] as wildcards and+\\ to quote a wildcard or backslash character literally. When GLOB+contains a `/', full pathnames are matched. Otherwise basenames+are matched. When GLOB ends with a `/', directories are included+as if \fB\-\-include\-dir\fR is specified. Otherwise files are included.+Note that \fB\-\-exclude\fR patterns take priority over \fB\-\-include\fR patterns.+GLOB should be quoted to prevent shell globbing. This option may+be repeated. .TP \fB\-\-include\-dir\fR=\fIGLOB\fR Only directories whose name matches GLOB are included in recursive searches, same as \fB\-g\fR \fIGLOB\fR/. GLOB can use **, *, ?, and [...] as-wildcards, and \\ to quote a wildcard or backslash character+wildcards and \\ to quote a wildcard or backslash character literally. When GLOB contains a `/', full pathnames are matched. Otherwise basenames are matched. Note that \fB\-\-exclude\-dir\fR patterns take priority over \fB\-\-include\-dir\fR patterns. GLOB should be quoted@@ -399,7 +408,7 @@ to prevent shell globbing. This option may be repeated. \fB\-\-include\-from\fR=\fIFILE\fR Read the globs from FILE and search only files and directories whose name matches one or more globs. A glob can use **, *, ?, and-[...] as wildcards, and \\ to quote a wildcard or backslash+[...] as wildcards and \\ to quote a wildcard or backslash character literally. When a glob contains a `/', full pathnames are matched. Otherwise basenames are matched. When a glob ends with a `/', directories are included as if \fB\-\-include\-dir\fR is@@ -505,14 +514,14 @@ each file processed. .TP \fB\-\-no\-group\-separator\fR Removes the group separator line from the output for context-options \fB\-A\fR, \fB\-B\fR, and \fB\-C\fR.+options \fB\-A\fR, \fB\-B\fR and \fB\-C\fR. .TP \fB\-\-not\fR [\fB\-e\fR] PATTERN Specifies that PATTERN should not match. Note that \fB\-e\fR \fIA\fR \fB\-\-not\fR \fB\-e\fR \fIB\fR matches lines with `A' or lines without a `B'. To match lines with `A' that have no `B', specify \fB\-e\fR \fIA\fR \fB\-\-andnot\fR \fB\-e\fR \fIB\fR. Option \fB\-\-stats\fR displays the search patterns applied. See also options \fB\-\-and\fR,-\fB\-\-andnot\fR, \fB\-\-bool\fR, \fB\-\-files\fR, and \fB\-\-lines\fR.+\fB\-\-andnot\fR, \fB\-\-bool\fR, \fB\-\-files\fR and \fB\-\-lines\fR. .TP \fB\-O\fR \fIEXTENSIONS\fR, \fB\-\-file\-extension\fR=\fIEXTENSIONS\fR Search only files whose filename extensions match the specified@@ -527,7 +536,7 @@ Print only the matching part of lines. When multiple lines match, the line numbers with option \fB\-n\fR are displayed using `|' as the field separator for each additional line matched by the pattern. If \fB\-u\fR is specified, ungroups multiple matches on the same line.-This option cannot be combined with options \fB\-A\fR, \fB\-B\fR, \fB\-C\fR, \fB\-v\fR, and \fB\-y\fR.+This option cannot be combined with options \fB\-A\fR, \fB\-B\fR, \fB\-C\fR, \fB\-v\fR and \fB\-y\fR. .TP \fB\-\-only\-line\-number\fR The line number of the matching line in the file is output without@@ -596,6 +605,13 @@ Recursively read all files under each directory, following symbolic links only if they are on the command line. When \fB\-J\fR1 is specified, files are searched in the same order as specified. .TP+\fB\-\-replace\fR=\fIFORMAT\fR+Replace matching patterns in the output by the specified FORMAT+with `%' fields. When option \fB\-P\fR is used, FORMAT may include `%1'+to `%9', `%[\fINUM\fR]#' and `%[\fINAME\fR]#' to output group captures. A `%%'+outputs `%' and `%~' outputs a newline. See option \fB\-\-format\fR,+`ugrep \fB\-\-help\fR format' and `man ugrep' section FORMAT for details.+.TP \fB\-S\fR, \fB\-\-dereference\fR If \fB\-r\fR is specified, all symbolic links are followed, like \fB\-R\fR. The default is not to follow symbolic links.@@ -610,7 +626,7 @@ a `\-', write the configuration to standard output. .TP \fB\-\-separator\fR[=\fISEP\fR] Use SEP as field separator between file name, line number, column-number, byte offset, and the matched line. The default is a colon+number, byte offset and the matched line. The default is a colon (`:'). .TP \fB\-\-sort\fR[=\fIKEY\fR]@@ -619,7 +635,7 @@ searches. KEY can be `name' to sort by pathname (default), `best' to sort by best match with option \fB\-Z\fR (sort by best match requires two passes over the input files), `size' to sort by file size, `used' to sort by last access time, `changed' to sort by last-modification time, and `created' to sort by creation time. Sorting+modification time and `created' to sort by creation time. Sorting is reversed with `rname', `rbest', `rsize', `rused', `rchanged', or `rcreated'. Archive contents are not sorted. Subdirectories are sorted and displayed after matching files. FILE arguments are@@ -627,12 +643,12 @@ searched in the same order as specified. Normally ugrep displays matches in no particular order to improve performance. .TP \fB\-\-stats\fR-Output statistics on the number of files and directories searched,+Output statistics on the number of files and directories searched and the inclusion and exclusion constraints applied. .TP \fB\-T\fR, \fB\-\-initial\-tab\fR Add a tab space to separate the file name, line number, column-number, and byte offset with the matched line.+number and byte offset with the matched line. .TP \fB\-t\fR \fITYPES\fR, \fB\-\-file\-type\fR=\fITYPES\fR Search only files associated with TYPES, a comma\-separated list of@@ -678,7 +694,7 @@ Output the matched line again for each additional pattern match, using `+' as the field separator. .TP \fB\-V\fR, \fB\-\-version\fR-Display version information and exit.+Display version with linked libraries and exit. .TP \fB\-v\fR, \fB\-\-invert\-match\fR Selected lines are those not matching any of the specified@@ -694,9 +710,9 @@ This option is equivalent to the \fB\-\-binary\-files\fR=with\-hex option. \fB\-w\fR, \fB\-\-word\-regexp\fR The PATTERN is searched for as a word, such that the matching text is preceded by a non\-word character and is followed by a non\-word-character. Word characters are letters, digits, and the+character. Word characters are letters, digits and the underscore. With option \fB\-P\fR, word characters are Unicode letters,-digits, and underscore. This option has no effect if \fB\-x\fR is also+digits and underscore. This option has no effect if \fB\-x\fR is also specified. If a PATTERN is specified, or \fB\-e\fR \fIPATTERN\fR or \fB\-N\fR \fIPATTERN\fR, then this option has no effect on \fB\-f\fR \fIFILE\fR patterns to allow \fB\-f\fR \fIFILE\fR patterns to narrow or widen the scope of the PATTERN search.@@ -724,7 +740,7 @@ not only lines containing the character `x'. .TP \fB\-y\fR, \fB\-\-any\-line\fR Any line is output (passthru). Non\-matching lines are output as-context with a `\-' separator. See also options \fB\-A\fR, \fB\-B\fR, and \fB\-C\fR.+context with a `\-' separator. See also options \fB\-A\fR, \fB\-B\fR and \fB\-C\fR. .TP \fB\-Z\fR[[+\-~]MAX], \fB\-\-fuzzy\fR[=[+\-~]MAX] Fuzzy mode: report approximate pattern matches within MAX errors.@@ -753,14 +769,14 @@ zstd (requires suffix .zst, .zstd, .tzst). .TP \fB\-\-zmax\fR=\fINUM\fR When used with option \fB\-z\fR (\fB\-\-decompress\fR), searches the contents of-compressed files and archives stored within archives for up to NUM-recursive levels deep. The default \fB\-\-zmax\fR=1 only permits searching+compressed files and archives stored within archives by up to NUM+recursive expansions. The default \fB\-\-zmax\fR=1 only permits searching uncompressed files stored in cpio, pax, tar and zip archives; compressed files and archives are detected as binary files and are effectively ignored. Specify \fB\-\-zmax\fR=2 to search compressed files and archives stored in cpio, pax, tar and zip archives. NUM may range from 1 to 99 for up to 99 decompression and de\-archiving-steps. Larger NUM values gradually degrade performance.+steps. Increasing NUM values gradually degrades performance. .TP \fB\-0\fR, \fB\-\-null\fR Prints a zero\-byte (NUL) after the file name. This option can be
@@ -30,7 +30,7 @@ @file cnf.cpp @brief CNF class for normalization of Boolean search queries @author Robert van Engelen - engelen@genivia.com-@copyright (c) 2019-2020, Robert van Engelen, Genivia Inc. All rights reserved.+@copyright (c) 2019-2022, Robert van Engelen, Genivia Inc. All rights reserved. @copyright (c) BSD-3 License - see LICENSE.txt */
@@ -30,7 +30,7 @@ @file cnf.hpp @brief CNF class for normalization of Boolean search queries @author Robert van Engelen - engelen@genivia.com-@copyright (c) 2019-2020, Robert van Engelen, Genivia Inc. All rights reserved.+@copyright (c) 2019-2022, Robert van Engelen, Genivia Inc. All rights reserved. @copyright (c) BSD-3 License - see LICENSE.txt */
@@ -30,7 +30,7 @@ @file flag.hpp @brief Tri-state Flag class and global flags set by options @author Robert van Engelen - engelen@genivia.com-@copyright (c) 2019-2020, Robert van Engelen, Genivia Inc. All rights reserved.+@copyright (c) 2019-2022, Robert van Engelen, Genivia Inc. All rights reserved. @copyright (c) BSD-3 License - see LICENSE.txt */ @@ -166,12 +166,13 @@ extern const char *flag_group_separator; extern const char *flag_hexdump; extern const char *flag_label; extern const char *flag_pager;-extern const char *flag_view;+extern const char *flag_replace; extern const char *flag_save_config; extern const char *flag_separator; extern const char *flag_sort; extern const char *flag_stats; extern const char *flag_tag;+extern const char *flag_view; extern std::string flag_config_file; extern std::set<std::string> flag_config_options; extern std::vector<std::string> flag_regexp;
@@ -30,7 +30,7 @@ @file mmap.hpp @brief class to manage memory-mapped files @author Robert van Engelen - engelen@genivia.com-@copyright (c) 2019-2020, Robert van Engelen, Genivia Inc. All rights reserved.+@copyright (c) 2019-2022, Robert van Engelen, Genivia Inc. All rights reserved. @copyright (c) BSD-3 License - see LICENSE.txt */
@@ -30,7 +30,7 @@ @file output.cpp @brief Output management @author Robert van Engelen - engelen@genivia.com-@copyright (c) 2019-2020, Robert van Engelen, Genivia Inc. All rights reserved.+@copyright (c) 2019-2022, Robert van Engelen, Genivia Inc. All rights reserved. @copyright (c) BSD-3 License - see LICENSE.txt */ @@ -407,14 +407,13 @@ std::pair<const char*,size_t> Output::capture(reflex::AbstractMatcher *matcher, while (true) { const char *bar = strchr(arg, '|');+ const char *end = strchr(arg, ']'); - if (bar == NULL)- {- bar = strchr(arg, ']');+ if (end == NULL)+ break; - if (bar == NULL)- break;- }+ if (bar == NULL || bar > end)+ bar = end; if (isdigit(*arg)) {@@ -728,20 +727,25 @@ void Output::format(const char *format, const char *pathname, const std::string& { size_t n = id.first; const char *bar;+ const char *end; while (true) { bar = strchr(arg, '|');+ end = strchr(arg, ']');++ if (bar == NULL || (end != NULL && bar > end))+ {+ bar = end;+ break;+ } - if (--n == 0 || bar == NULL)+ if (--n == 0) break; arg = bar + 1; } - if (bar == NULL)- bar = strchr(arg, ']');- if (n == 0 && bar != NULL) str(arg, bar - arg); else if (id.second != NULL)@@ -771,20 +775,25 @@ void Output::format(const char *format, const char *pathname, const std::string& if (n > 0) { const char *bar;+ const char *end; while (true) { bar = strchr(arg, '|');+ end = strchr(arg, ']'); - if (--n == 0 || bar == NULL)+ if (bar == NULL || (end != NULL && bar > end))+ {+ bar = end;+ break;+ }++ if (--n == 0) break; arg = bar + 1; } - if (bar == NULL)- bar = strchr(arg, ']');- if (n == 0 && bar != NULL) str(arg, bar - arg); else if (id.second != NULL)
@@ -30,7 +30,7 @@ @file output.hpp @brief Output management @author Robert van Engelen - engelen@genivia.com-@copyright (c) 2019-2020, Robert van Engelen, Genivia Inc. All rights reserved.+@copyright (c) 2019-2022, Robert van Engelen, Genivia Inc. All rights reserved. @copyright (c) BSD-3 License - see LICENSE.txt */
@@ -30,7 +30,7 @@ @file query.cpp @brief Query engine and UI @author Robert van Engelen - engelen@genivia.com-@copyright (c) 2019-2020, Robert van Engelen, Genivia Inc. All rights reserved.+@copyright (c) 2019-2022, Robert van Engelen, Genivia Inc. All rights reserved. @copyright (c) BSD-3 License - see LICENSE.txt */ @@ -1341,7 +1341,9 @@ void Query::search() // clear "Searching..." or "(END)", if displayed if (error_ == -1 && rows_ < row_ + Screen::rows - 1) {- Screen::setpos(rows_ - row_ + 1, 0);+ Screen::normal();+ Screen::invert();+ Screen::put(rows_ - row_ + 1, 0, "Restarting search..."); Screen::normal(); Screen::erase(); }@@ -2924,19 +2926,19 @@ void Query::meta(int key) { flags_[30].flag = true; - if (fuzzy_ > 1)+ if ((fuzzy_ & 0xff) > 1) --fuzzy_; - msg.append(" to ").append(std::to_string(fuzzy_));+ msg.append(" to ").append(std::to_string(fuzzy_ & 0xff)); } else if (key == ']') {- if (flags_[30].flag)+ if (flags_[30].flag && (fuzzy_ & 0xff) < 0xff) ++fuzzy_; else flags_[30].flag = true; - msg.append(" to ").append(std::to_string(fuzzy_));+ msg.append(" to ").append(std::to_string(fuzzy_ & 0xff)); } else if (key == '{') {
@@ -30,7 +30,7 @@ @file query.hpp @brief Query engine and UI @author Robert van Engelen - engelen@genivia.com-@copyright (c) 2019-2020, Robert van Engelen, Genivia Inc. All rights reserved.+@copyright (c) 2019-2022, Robert van Engelen, Genivia Inc. All rights reserved. @copyright (c) BSD-3 License - see LICENSE.txt */
@@ -30,7 +30,7 @@ @file screen.cpp @brief ANSI SGR code controlled screen API - static, not thread safe @author Robert van Engelen - engelen@genivia.com-@copyright (c) 2019-2020, Robert van Engelen, Genivia Inc. All rights reserved.+@copyright (c) 2019-2022, Robert van Engelen, Genivia Inc. All rights reserved. @copyright (c) BSD-3 License - see LICENSE.txt */
@@ -30,7 +30,7 @@ @file screen.hpp @brief ANSI SGR code controlled screen API - static, not thread safe @author Robert van Engelen - engelen@genivia.com-@copyright (c) 2019-2020, Robert van Engelen, Genivia Inc. All rights reserved.+@copyright (c) 2019-2022, Robert van Engelen, Genivia Inc. All rights reserved. @copyright (c) BSD-3 License - see LICENSE.txt */
@@ -30,7 +30,7 @@ @file stats.cpp @brief collect global statistics - static, partially thread-safe @author Robert van Engelen - engelen@genivia.com-@copyright (c) 2019-2020, Robert van Engelen, Genivia Inc. All rights reserved.+@copyright (c) 2019-2022, Robert van Engelen, Genivia Inc. All rights reserved. @copyright (c) BSD-3 License - see LICENSE.txt */
@@ -30,7 +30,7 @@ @file stats.hpp @brief collect global statistics - static, partially thread-safe (see below) @author Robert van Engelen - engelen@genivia.com-@copyright (c) 2019-2020, Robert van Engelen, Genivia Inc. All rights reserved.+@copyright (c) 2019-2022, Robert van Engelen, Genivia Inc. All rights reserved. @copyright (c) BSD-3 License - see LICENSE.txt Thread-safe functions:
@@ -30,7 +30,7 @@ @file ugrep.cpp @brief a pattern search utility written in C++11 @author Robert van Engelen - engelen@genivia.com-@copyright (c) 2019-2021, Robert van Engelen, Genivia Inc. All rights reserved.+@copyright (c) 2019-2022, Robert van Engelen, Genivia Inc. All rights reserved. @copyright (c) BSD-3 License - see LICENSE.txt For download and installation instructions:@@ -404,12 +404,13 @@ const char *flag_group_separator = "--"; const char *flag_hexdump = NULL; const char *flag_label = "(standard input)"; const char *flag_pager = DEFAULT_PAGER;-const char *flag_view = "";+const char *flag_replace = NULL; const char *flag_save_config = NULL; const char *flag_separator = ":"; const char *flag_sort = NULL; const char *flag_stats = NULL; const char *flag_tag = NULL;+const char *flag_view = ""; std::string flag_config_file; std::set<std::string> flag_config_options; std::vector<std::string> flag_regexp;@@ -2160,7 +2161,7 @@ struct Grep { }; - // extend event AnyLineGrepHandler to output specific context lines for -A, -B, and -C+ // extend event AnyLineGrepHandler to output specific context lines for -A, -B and -C struct ContextGrepHandler : public AnyLineGrepHandler { // context state to track context lines before and after a match@@ -2398,7 +2399,7 @@ struct Grep { }; - // extend event AnyLineGrepHandler to output specific context lines for -A, -B, and -C with -v+ // extend event AnyLineGrepHandler to output specific context lines for -A, -B and -C with -v struct InvertContextGrepHandler : public AnyLineGrepHandler { struct InvertContextMatch {@@ -3336,12 +3337,10 @@ struct Grep { return true; } - // after opening a file with init_read, check if it is binary+ // after opening a file with init_read, check if its initial part (64K or what could be read) is binary bool init_is_binary() {- // limit checking to first buffer filled with input up to 16K, which should suffice, to improve performance- size_t avail = matcher->avail();- return is_binary(matcher->begin(), avail < 16384 ? avail : 16384);+ return is_binary(matcher->begin(), matcher->avail()); } const char *filename; // the name of the file being searched@@ -4061,7 +4060,7 @@ static void load_config(std::list<std::pair<CNF::PATTERN,const char*>>& pattern_ if (errors) {- std::cerr << "Try 'ugrep --help [WHAT]' for more information\n";+ std::cerr << "Try `ugrep --help' or `ugrep --help WHAT' for more information\n"; exit(EXIT_ERROR); }@@ -4098,7 +4097,7 @@ static void save_config() # A long option is defined per line with an optional `=' and its argument,\n\ # when applicable. Empty lines and lines starting with a `#' are ignored.\n\ #\n\-# Try `ugrep --help [WHAT]' for more information.\n\n");+# Try `ugrep --help' or `ugrep --help WHAT' for more information.\n\n"); fprintf(file, "### TERMINAL DISPLAY ###\n\n"); @@ -4659,10 +4658,12 @@ void options(std::list<std::pair<CNF::PATTERN,const char*>>& pattern_args, int a flag_directories = "recurse"; else if (strncmp(arg, "regexp=", 7) == 0) option_regexp(pattern_args, arg + 7);- else if (strcmp(arg, "range") == 0)+ else if (strncmp(arg, "replace=", 8) == 0)+ flag_replace = arg + 8;+ else if (strcmp(arg, "range") == 0 || strcmp(arg, "regexp") == 0 || strcmp(arg, "replace") == 0) usage("missing argument for --", arg); else- usage("invalid option --", arg, "--range, --recursive or --regexp");+ usage("invalid option --", arg, "--range, --recursive, --regexp or --replace"); break; case 's':@@ -5376,7 +5377,7 @@ void init(int argc, const char **argv) if (warnings > 0) { std::cerr << "Usage: ugrep [OPTIONS] [PATTERN] [-f FILE] [-e PATTERN] [FILE ...]\n";- std::cerr << "Try 'ugrep --help [WHAT]' for more information\n";+ std::cerr << "Try `ugrep --help' or `ugrep --help WHAT' for more information\n"; exit(EXIT_ERROR); } @@ -5430,7 +5431,7 @@ void init(int argc, const char **argv) if (arg.first == CNF::PATTERN::TERM) bcnf.new_term(); else- bcnf.new_pattern(arg.first, arg.second); // relies on options --bool, -F, -G, -w, -x, and -f+ bcnf.new_pattern(arg.first, arg.second); // relies on options --bool, -F, -G, -w, -x and -f } // --query: override --pager@@ -5495,7 +5496,7 @@ void init(int argc, const char **argv) else if (strcmp(flag_binary_files, "with-hex") == 0) flag_with_hex = true; else if (strcmp(flag_binary_files, "binary") != 0)- usage("invalid argument --binary-files=TYPE, valid arguments are 'binary', 'without-match', 'text', 'hex', and 'with-hex'");+ usage("invalid argument --binary-files=TYPE, valid arguments are 'binary', 'without-match', 'text', 'hex' and 'with-hex'"); // --hex takes priority over --with-hex takes priority over -I takes priority over -a if (flag_hex)@@ -5640,7 +5641,7 @@ void init(int argc, const char **argv) else if (strcmp(flag_directories, "recurse") == 0) flag_directories_action = Action::RECURSE; else- usage("invalid argument -d ACTION, valid arguments are 'skip', 'read', 'recurse', and 'dereference-recurse'");+ usage("invalid argument -d ACTION, valid arguments are 'skip', 'read', 'recurse' and 'dereference-recurse'"); // if no FILE specified and no -r or -R specified, when reading standard input from a TTY then enable -R if (!flag_stdin && arg_files.empty() && flag_directories_action != Action::RECURSE && isatty(STDIN_FILENO))@@ -5771,6 +5772,14 @@ void init(int argc, const char **argv) flag_format_end = "</grep>\n"; } + // --replace clashes with --format+ if (flag_replace != NULL && flag_format != NULL)+ abort("--format is not permitted with --replace");++ // -v with --files is not permitted+ if (flag_files && flag_invert_match)+ abort("-v is not permitted with --files, invert the Boolean query instead");+ #ifdef HAVE_STATVFS // --exclude-fs: add file system ids to exclude@@ -6239,7 +6248,7 @@ void terminal() } else if (strcmp(flag_apply_color, "always") != 0 && strcmp(flag_apply_color, "yes") != 0 && strcmp(flag_apply_color, "force") != 0) {- usage("invalid argument --color=WHEN, valid arguments are 'never', 'always', and 'auto'");+ usage("invalid argument --color=WHEN, valid arguments are 'never', 'always' and 'auto'"); } if (flag_apply_color != NULL)@@ -6547,7 +6556,7 @@ void ugrep() else if (strcmp(flag_sort, "created") == 0 || strcmp(flag_sort, "rcreated") == 0) flag_sort_key = Sort::CREATED; else- usage("invalid argument --sort=KEY, valid arguments are 'name', 'best', 'size', 'used', 'changed', 'created', 'rname', 'rbest', 'rsize', 'rused', 'rchanged', and 'rcreated'");+ usage("invalid argument --sort=KEY, valid arguments are 'name', 'best', 'size', 'used', 'changed', 'created', 'rname', 'rbest', 'rsize', 'rused', 'rchanged' and 'rcreated'"); } // add PATTERN to the CNF@@ -6603,13 +6612,6 @@ void ugrep() } } - // -v with --files is not permitted- if (flag_files && flag_invert_match)- {- abort("-v is not permitted with --files, invert the Boolean query instead");- flag_invert_match = false;- }- // -x or --match: enable -Y and disable --dotall and -w if (flag_line_regexp || flag_match) {@@ -6781,7 +6783,7 @@ void ugrep() } } - // -y: disable -A, -B, and -C+ // -y: disable -A, -B and -C if (flag_any_line) flag_after_context = flag_before_context = 0; @@ -8491,7 +8493,7 @@ void Grep::search(const char *pathname) continue; } - // output --format-open+ // --format-open if (matches == 0) { if (flag_files && matchers != NULL)@@ -8519,6 +8521,7 @@ void Grep::search(const char *pathname) if (flag_format_open != NULL) {+ // output --format-open=FORMAT out.format(flag_format_open, pathname, partname, Stats::found_parts(), matcher, false, Stats::found_parts() > 1); // --files: undo files count@@ -8529,7 +8532,7 @@ void Grep::search(const char *pathname) ++matches; - // output --format+ // output --format=FORMAT out.format(flag_format, pathname, partname, matches, matcher, matches > 1, matches > 1); // -m: max number of matches reached?@@ -8545,7 +8548,7 @@ void Grep::search(const char *pathname) if (!cnf_satisfied(true)) goto exit_search; - // output --format-close+ // output --format-close=FORMAT if (matches > 0 && flag_format_close != NULL) out.format(flag_format_close, pathname, partname, Stats::found_parts(), matcher, false, Stats::found_parts() > 1); }@@ -8606,7 +8609,7 @@ void Grep::search(const char *pathname) // option -o size_t lineno = 0;- bool binfile = !flag_text && !flag_hex && !flag_with_hex && init_is_binary();+ bool binfile = !flag_text && !flag_hex && !flag_with_hex && !flag_binary_without_match && init_is_binary(); bool hex = false; bool nl = false; @@ -8723,6 +8726,13 @@ void Grep::search(const char *pathname) lineno += matcher->lines() - 1; }+ else if (flag_replace != NULL)+ {+ // output --replace=FORMAT+ out.str(match_ms);+ out.format(flag_replace, pathname, partname, matches, matcher, matches > 1, matches > 1);+ out.str(match_off);+ } else { // echo multi-line matches line-by-line@@ -8774,7 +8784,7 @@ void Grep::search(const char *pathname) // options -A, -B, -C, -y, -v are not specified size_t lineno = 0;- bool binfile = !flag_text && !flag_hex && !flag_with_hex && init_is_binary();+ bool binfile = !flag_text && !flag_hex && !flag_with_hex && !flag_binary_without_match && init_is_binary(); bool hex = false; bool binary = false; const char *restline_data = NULL;@@ -8906,29 +8916,39 @@ void Grep::search(const char *pathname) out.str(bol, border); out.str(color_off); - // echo multi-line matches line-by-line-- const char *from = begin;- const char *to;-- while ((to = static_cast<const char*>(memchr(from, '\n', size - (from - begin)))) != NULL)+ if (flag_replace != NULL) {+ // output --replace=FORMAT out.str(match_ms);- out.str(from, to - from);+ out.format(flag_replace, pathname, partname, matches, matcher, matches > 1, matches > 1); out.str(match_off);- out.chr('\n');+ }+ else+ {+ // echo multi-line matches line-by-line - out.header(pathname, partname, ++lineno, NULL, first + (to - begin) + 1, "|", false);+ const char *from = begin;+ const char *to; - from = to + 1;- }+ while ((to = static_cast<const char*>(memchr(from, '\n', size - (from - begin)))) != NULL)+ {+ out.str(match_ms);+ out.str(from, to - from);+ out.str(match_off);+ out.chr('\n'); - size -= from - begin;- begin = from;+ out.header(pathname, partname, ++lineno, NULL, first + (to - begin) + 1, "|", false); - out.str(match_ms);- out.str(begin, size);- out.str(match_off);+ from = to + 1;+ }++ size -= from - begin;+ begin = from;++ out.str(match_ms);+ out.str(begin, size);+ out.str(match_off);+ } if (flag_ungroup) {@@ -8987,34 +9007,44 @@ void Grep::search(const char *pathname) out.str(restline_data, first - restline_last); out.str(color_off); - if (lines > 1)+ if (flag_replace != NULL) {- // echo multi-line matches line-by-line+ // output --replace=FORMAT+ out.str(match_ms);+ out.format(flag_replace, pathname, partname, matches, matcher, matches > 1, matches > 1);+ out.str(match_off);+ }+ else+ {+ if (lines > 1)+ {+ // echo multi-line matches line-by-line - const char *from = begin;- const char *to;- size_t num = 1;+ const char *from = begin;+ const char *to;+ size_t num = 1; - while ((to = static_cast<const char*>(memchr(from, '\n', size - (from - begin)))) != NULL)- {- out.str(match_ms);- out.str(from, to - from);- out.str(match_off);- out.chr('\n');+ while ((to = static_cast<const char*>(memchr(from, '\n', size - (from - begin)))) != NULL)+ {+ out.str(match_ms);+ out.str(from, to - from);+ out.str(match_off);+ out.chr('\n'); - out.header(pathname, partname, lineno + num, NULL, first + (to - begin) + 1, "|", false);+ out.header(pathname, partname, lineno + num, NULL, first + (to - begin) + 1, "|", false); - from = to + 1;- ++num;+ from = to + 1;+ ++num;+ }++ size -= from - begin;+ begin = from; } - size -= from - begin;- begin = from;+ out.str(match_ms);+ out.str(begin, size);+ out.str(match_off); }-- out.str(match_ms);- out.str(begin, size);- out.str(match_off); } if (lines == 1)@@ -9121,7 +9151,7 @@ void Grep::search(const char *pathname) // InvertMatchHandler requires lineno to be set precisely, i.e. after skipping --range lines size_t lineno = flag_min_line > 0 ? flag_min_line - 1 : 0;- bool binfile = !flag_text && !flag_hex && !flag_with_hex && init_is_binary();+ bool binfile = !flag_text && !flag_hex && !flag_with_hex && !flag_binary_without_match && init_is_binary(); bool hex = false; bool binary = false; bool stop = false;@@ -9207,7 +9237,7 @@ void Grep::search(const char *pathname) // AnyLineGrepHandler requires lineno to be set precisely, i.e. after skipping --range lines size_t lineno = flag_min_line > 0 ? flag_min_line - 1 : 0;- bool binfile = !flag_text && !flag_hex && !flag_with_hex && init_is_binary();+ bool binfile = !flag_text && !flag_hex && !flag_with_hex && !flag_binary_without_match && init_is_binary(); bool hex = false; bool binary = false; bool stop = false;@@ -9395,29 +9425,39 @@ void Grep::search(const char *pathname) out.str(bol, border); out.str(color_off); - // echo multi-line matches line-by-line-- const char *from = begin;- const char *to;-- while ((to = static_cast<const char*>(memchr(from, '\n', size - (from - begin)))) != NULL)+ if (flag_replace != NULL) {+ // output --replace=FORMAT out.str(v_match_ms);- out.str(from, to - from);+ out.format(flag_replace, pathname, partname, matches, matcher, matches > 1, matches > 1); out.str(match_off);- out.chr('\n');+ }+ else+ {+ // echo multi-line matches line-by-line - out.header(pathname, partname, ++lineno, NULL, first + (to - begin) + 1, "|", false);+ const char *from = begin;+ const char *to; - from = to + 1;- }+ while ((to = static_cast<const char*>(memchr(from, '\n', size - (from - begin)))) != NULL)+ {+ out.str(v_match_ms);+ out.str(from, to - from);+ out.str(match_off);+ out.chr('\n'); - size -= from - begin;- begin = from;+ out.header(pathname, partname, ++lineno, NULL, first + (to - begin) + 1, "|", false); - out.str(v_match_ms);- out.str(begin, size);- out.str(match_off);+ from = to + 1;+ }++ size -= from - begin;+ begin = from;++ out.str(v_match_ms);+ out.str(begin, size);+ out.str(match_off);+ } if (flag_ungroup) {@@ -9476,34 +9516,44 @@ void Grep::search(const char *pathname) out.str(restline_data, first - restline_last); out.str(color_off); - if (lines > 1)+ if (flag_replace != NULL) {- // echo multi-line matches line-by-line+ // output --replace=FORMAT+ out.str(v_match_ms);+ out.format(flag_replace, pathname, partname, matches, matcher, matches > 1, matches > 1);+ out.str(match_off);+ }+ else+ {+ if (lines > 1)+ {+ // echo multi-line matches line-by-line - const char *from = begin;- const char *to;- size_t num = 1;+ const char *from = begin;+ const char *to;+ size_t num = 1; - while ((to = static_cast<const char*>(memchr(from, '\n', size - (from - begin)))) != NULL)- {- out.str(v_match_ms);- out.str(from, to - from);- out.str(match_off);- out.chr('\n');+ while ((to = static_cast<const char*>(memchr(from, '\n', size - (from - begin)))) != NULL)+ {+ out.str(v_match_ms);+ out.str(from, to - from);+ out.str(match_off);+ out.chr('\n'); - out.header(pathname, partname, lineno + num, NULL, first + (to - begin) + 1, "|", false);+ out.header(pathname, partname, lineno + num, NULL, first + (to - begin) + 1, "|", false); - from = to + 1;- ++num;+ from = to + 1;+ ++num;+ }++ size -= from - begin;+ begin = from; } - size -= from - begin;- begin = from;+ out.str(v_match_ms);+ out.str(begin, size);+ out.str(match_off); }-- out.str(v_match_ms);- out.str(begin, size);- out.str(match_off); } if (lines == 1)@@ -9624,7 +9674,7 @@ void Grep::search(const char *pathname) // ContextGrepHandler requires lineno to be set precisely, i.e. after skipping --range lines size_t lineno = flag_min_line > 0 ? flag_min_line - 1 : 0;- bool binfile = !flag_text && !flag_hex && !flag_with_hex && init_is_binary();+ bool binfile = !flag_text && !flag_hex && !flag_with_hex && !flag_binary_without_match && init_is_binary(); bool hex = false; bool binary = false; bool stop = false;@@ -9798,29 +9848,39 @@ void Grep::search(const char *pathname) out.str(bol, border); out.str(color_off); - // echo multi-line matches line-by-line-- const char *from = begin;- const char *to;-- while ((to = static_cast<const char*>(memchr(from, '\n', size - (from - begin)))) != NULL)+ if (flag_replace != NULL) {+ // output --replace=FORMAT out.str(match_ms);- out.str(from, to - from);+ out.format(flag_replace, pathname, partname, matches, matcher, matches > 1, matches > 1); out.str(match_off);- out.chr('\n');+ }+ else+ {+ // echo multi-line matches line-by-line - out.header(pathname, partname, ++lineno, NULL, first + (to - begin) + 1, "|", false);+ const char *from = begin;+ const char *to; - from = to + 1;- }+ while ((to = static_cast<const char*>(memchr(from, '\n', size - (from - begin)))) != NULL)+ {+ out.str(match_ms);+ out.str(from, to - from);+ out.str(match_off);+ out.chr('\n'); - size -= from - begin;- begin = from;+ out.header(pathname, partname, ++lineno, NULL, first + (to - begin) + 1, "|", false); - out.str(match_ms);- out.str(begin, size);- out.str(match_off);+ from = to + 1;+ }++ size -= from - begin;+ begin = from;++ out.str(match_ms);+ out.str(begin, size);+ out.str(match_off);+ } if (flag_ungroup) {@@ -9879,34 +9939,44 @@ void Grep::search(const char *pathname) out.str(restline_data, first - restline_last); out.str(color_off); - if (lines > 1)+ if (flag_replace != NULL) {- // echo multi-line matches line-by-line+ // output --replace=FORMAT+ out.str(match_ms);+ out.format(flag_replace, pathname, partname, matches, matcher, matches > 1, matches > 1);+ out.str(match_off);+ }+ else+ {+ if (lines > 1)+ {+ // echo multi-line matches line-by-line - const char *from = begin;- const char *to;- size_t num = 1;+ const char *from = begin;+ const char *to;+ size_t num = 1; - while ((to = static_cast<const char*>(memchr(from, '\n', size - (from - begin)))) != NULL)- {- out.str(match_ms);- out.str(from, to - from);- out.str(match_off);- out.chr('\n');+ while ((to = static_cast<const char*>(memchr(from, '\n', size - (from - begin)))) != NULL)+ {+ out.str(match_ms);+ out.str(from, to - from);+ out.str(match_off);+ out.chr('\n'); - out.header(pathname, partname, lineno + num, NULL, first + (to - begin) + 1, "|", false);+ out.header(pathname, partname, lineno + num, NULL, first + (to - begin) + 1, "|", false); - from = to + 1;- ++num;+ from = to + 1;+ ++num;+ }++ size -= from - begin;+ begin = from; } - size -= from - begin;- begin = from;+ out.str(match_ms);+ out.str(begin, size);+ out.str(match_off); }-- out.str(match_ms);- out.str(begin, size);- out.str(match_off); } if (lines == 1)@@ -10031,7 +10101,7 @@ void Grep::search(const char *pathname) size_t lineno = flag_min_line > 0 ? flag_min_line - 1 : 0; size_t last_lineno = 0; size_t after = flag_after_context;- bool binfile = !flag_text && !flag_hex && !flag_with_hex && init_is_binary();+ bool binfile = !flag_text && !flag_hex && !flag_with_hex && !flag_binary_without_match && init_is_binary(); bool hex = false; bool binary = false; bool stop = false;@@ -10204,35 +10274,45 @@ void Grep::search(const char *pathname) out.str(bol, border); out.str(color_off); - if (lines > 1)+ if (flag_replace != NULL) {- // echo multi-line matches line-by-line+ // output --replace=FORMAT+ out.str(match_mc);+ out.format(flag_replace, pathname, partname, matches, matcher, matches > 1, matches > 1);+ out.str(match_off);+ }+ else+ {+ if (lines > 1)+ {+ // echo multi-line matches line-by-line - const char *from = begin;- const char *to;- size_t num = 1;+ const char *from = begin;+ const char *to;+ size_t num = 1; - while ((to = static_cast<const char*>(memchr(from, '\n', size - (from - begin)))) != NULL)- {- out.str(match_mc);- out.str(from, to - from);- out.str(match_off);- out.chr('\n');+ while ((to = static_cast<const char*>(memchr(from, '\n', size - (from - begin)))) != NULL)+ {+ out.str(match_mc);+ out.str(from, to - from);+ out.str(match_off);+ out.chr('\n'); - out.header(pathname, partname, lineno + num, NULL, first + (to - begin) + 1, "-", false);+ out.header(pathname, partname, lineno + num, NULL, first + (to - begin) + 1, "-", false); - from = to + 1;- ++num;+ from = to + 1;+ ++num;+ }++ size -= from - begin;+ begin = from; } - size -= from - begin;- begin = from;+ out.str(match_mc);+ out.str(begin, size);+ out.str(match_off); } - out.str(match_mc);- out.str(begin, size);- out.str(match_off);- restline.assign(end, eol - end); restline_data = restline.c_str(); restline_size = restline.size();@@ -10328,34 +10408,44 @@ void Grep::search(const char *pathname) out.str(restline_data, first - restline_last); out.str(color_off); - if (lines > 1)+ if (flag_replace != NULL) {- // echo multi-line matches line-by-line+ // output --replace=FORMAT+ out.str(match_mc);+ out.format(flag_replace, pathname, partname, matches, matcher, matches > 1, matches > 1);+ out.str(match_off);+ }+ else+ {+ if (lines > 1)+ {+ // echo multi-line matches line-by-line - const char *from = begin;- const char *to;- size_t num = 1;+ const char *from = begin;+ const char *to;+ size_t num = 1; - while ((to = static_cast<const char*>(memchr(from, '\n', size - (from - begin)))) != NULL)- {- out.str(match_mc);- out.str(from, to - from);- out.str(match_off);- out.chr('\n');+ while ((to = static_cast<const char*>(memchr(from, '\n', size - (from - begin)))) != NULL)+ {+ out.str(match_mc);+ out.str(from, to - from);+ out.str(match_off);+ out.chr('\n'); - out.header(pathname, partname, lineno + num, NULL, first + (to - begin) + 1, "-", false);+ out.header(pathname, partname, lineno + num, NULL, first + (to - begin) + 1, "-", false); - from = to + 1;- ++num;+ from = to + 1;+ ++num;+ }++ size -= from - begin;+ begin = from; } - size -= from - begin;- begin = from;+ out.str(match_mc);+ out.str(begin, size);+ out.str(match_off); }-- out.str(match_mc);- out.str(begin, size);- out.str(match_off); } if (lines == 1)@@ -10981,9 +11071,12 @@ void usage(const char *message, const char *arg, const char *valid) std::cerr << "ugrep: " << message << (arg != NULL ? arg : ""); if (valid != NULL) std::cerr << ", did you mean " << valid << "?";- std::cerr << std::endl;+ std::cerr << " For more help on options, try `ugrep --help' or `ugrep --help " << arg << "'" << std::endl;++ // do not exit when reading a config file if (!flag_usage_warnings) exit(EXIT_ERROR);+ ++warnings; } @@ -10995,7 +11088,7 @@ void help(std::ostream& out) -A NUM, --after-context=NUM\n\ Print NUM lines of trailing context after matching lines. Places\n\ a --group-separator between contiguous groups of matches. See also\n\- options -B, -C, and -y.\n\+ options -B, -C and -y.\n\ -a, --text\n\ Process a binary file as if it were text. This is equivalent to\n\ the --binary-files=text option. This option might output binary\n\@@ -11009,22 +11102,22 @@ void help(std::ostream& out) (`A' or `B') and (`C' or `D'). Note that multiple -e PATTERN are\n\ alternations that bind more tightly together than --and. Option\n\ --stats displays the search patterns applied. See also options\n\- --not, --andnot, --bool, --files, and --lines.\n\+ --not, --andnot, --bool, --files and --lines.\n\ --andnot [[-e] PATTERN] ...\n\- Combines --and --not. See also options --and, --not, and --bool.\n\+ Combines --and --not. See also options --and, --not and --bool.\n\ -B NUM, --before-context=NUM\n\ Print NUM lines of leading context before matching lines. Places\n\ a --group-separator between contiguous groups of matches. See also\n\- options -A, -C, and -y.\n\+ options -A, -C and -y.\n\ -b, --byte-offset\n\ The offset in bytes of a matched line is displayed in front of the\n\ respective matched line. If -u is specified, displays the offset\n\ for each pattern matched on the same line. Byte offsets are exact\n\- for ASCII, UTF-8, and raw binary input. Otherwise, the byte offset\n\+ for ASCII, UTF-8 and raw binary input. Otherwise, the byte offset\n\ in the UTF-8 normalized input is displayed.\n\ --binary-files=TYPE\n\ Controls searching and reporting pattern matches in binary files.\n\- TYPE can be `binary', `without-match`, `text`, `hex`, and\n\+ TYPE can be `binary', `without-match`, `text`, `hex` and\n\ `with-hex'. The default is `binary' to search binary files and to\n\ report a match without displaying the match. `without-match'\n\ ignores binary matches. `text' treats all binary files as text,\n\@@ -11033,12 +11126,12 @@ void help(std::ostream& out) it as commands. `hex' reports all matches in hexadecimal.\n\ `with-hex' only reports binary matches in hexadecimal, leaving text\n\ matches alone. A match is considered binary when matching a zero\n\- byte or invalid UTF. Short options are -a, -I, -U, -W, and -X.\n\+ byte or invalid UTF. Short options are -a, -I, -U, -W and -X.\n\ --bool, -%\n\ Specifies Boolean query patterns. A Boolean query pattern is\n\ composed of `AND', `OR', `NOT' operators and grouping with `(' `)'.\n\ Spacing between subpatterns is the same as `AND', `|' is the same\n\- as `OR', and a `-' is the same as `NOT'. The `OR' operator binds\n\+ as `OR' and a `-' is the same as `NOT'. The `OR' operator binds\n\ more tightly than `AND'. For example, --bool 'A|B C|D' matches\n\ lines with (`A' or `B') and (`C' or `D'), --bool 'A -B' matches\n\ lines with `A' and not `B'. Operators `AND', `OR', `NOT' require\n\@@ -11050,17 +11143,17 @@ void help(std::ostream& out) for grouping. For example, --bool '(A B)|C' matches lines with `A'\n\ and `B', or lines with `C'. Note that all subpatterns in a Boolean\n\ query pattern are regular expressions, unless option -F is used.\n\- Options -E, -F, -G, -P, and -Z can be combined with --bool to match\n\+ Options -E, -F, -G, -P and -Z can be combined with --bool to match\n\ subpatterns as strings or regular expressions (-E is the default.)\n\ This option does not apply to -f FILE patterns. Option --stats\n\ displays the search patterns applied. See also options --and,\n\- --andnot, --not, --files, and --lines.\n\+ --andnot, --not, --files and --lines.\n\ --break\n\ Adds a line break between results from different files.\n\ -C NUM, --context=NUM\n\ Print NUM lines of leading and trailing context surrounding each\n\ match. Places a --group-separator between contiguous groups of\n\- matches. See also options -A, -B, and -y.\n\+ matches. See also options -A, -B and -y.\n\ -c, --count\n\ Only a count of selected lines is written to standard output.\n\ If -o or -u is specified, counts the number of patterns matched.\n\@@ -11136,7 +11229,7 @@ void help(std::ostream& out) out << ".\n\ --exclude=GLOB\n\ Skip files whose name matches GLOB using wildcard matching, same as\n\- -g ^GLOB. GLOB can use **, *, ?, and [...] as wildcards, and \\ to\n\+ -g ^GLOB. GLOB can use **, *, ?, and [...] as wildcards and \\ to\n\ quote a wildcard or backslash character literally. When GLOB\n\ contains a `/', full pathnames are matched. Otherwise basenames\n\ are matched. When GLOB ends with a `/', directories are excluded\n\@@ -11147,7 +11240,7 @@ void help(std::ostream& out) --exclude-dir=GLOB\n\ Exclude directories whose name matches GLOB from recursive\n\ searches, same as -g ^GLOB/. GLOB can use **, *, ?, and [...] as\n\- wildcards, and \\ to quote a wildcard or backslash character\n\+ wildcards and \\ to quote a wildcard or backslash character\n\ literally. When GLOB contains a `/', full pathnames are matched.\n\ Otherwise basenames are matched. Note that --exclude-dir patterns\n\ take priority over --include-dir patterns. GLOB should be quoted\n\@@ -11155,7 +11248,7 @@ void help(std::ostream& out) --exclude-from=FILE\n\ Read the globs from FILE and skip files and directories whose name\n\ matches one or more globs. A glob can use **, *, ?, and [...] as\n\- wildcards, and \\ to quote a wildcard or backslash character\n\+ wildcards and \\ to quote a wildcard or backslash character\n\ literally. When a glob contains a `/', full pathnames are matched.\n\ Otherwise basenames are matched. When a glob ends with a `/',\n\ directories are excluded as if --exclude-dir is specified.\n\@@ -11216,8 +11309,11 @@ void help(std::ostream& out) --format=FORMAT\n\ Output FORMAT-formatted matches. For example --format='%f:%n:%O%~'\n\ outputs matching lines `%O' with filename `%f` and line number `%n'\n\- followed by a newline `%~'. Context options -A, -B, -C, and -y are\n\- ignored. See `man ugrep' section FORMAT.\n\+ followed by a newline `%~'. When option -P is used, FORMAT may\n\+ include `%1' to `%9', `%[NUM]#' and `%[NAME]#' to output group\n\+ captures. A `%%' outputs `%'. See `ugrep --help format' and `man\n\+ ugrep' section FORMAT for details. Context options -A, -B, -C and\n\+ -y are ignored.\n\ --free-space\n\ Spacing (blanks and tabs) in regular expressions are ignored.\n\ -G, --basic-regexp\n\@@ -11235,7 +11331,7 @@ void help(std::ostream& out) be combined with options -M, -O and -t to expand the recursive\n\ search.\n\ --group-separator[=SEP]\n\- Use SEP as a group separator for context options -A, -B, and -C.\n\+ Use SEP as a group separator for context options -A, -B and -C.\n\ The default is a double hyphen (`--').\n\ -H, --with-filename\n\ Always print the filename with output lines. This is the default\n\@@ -11248,6 +11344,9 @@ void help(std::ostream& out) results from different files.\n\ --help [WHAT], -? [WHAT]\n\ Display a help message, specifically on WHAT when specified.\n\+ In addition, `--help format' displays an overview of FORMAT fields,\n\+ `--help regex' displays an overview of regular expressions and\n\+ `--help globs' displays an overview of glob syntax and conventions.\n\ --hexdump=[1-8][a][b][c][h]\n\ Output matches in 1 to 8 columns of 8 hexadecimal octets. The\n\ default is 2 columns or 16 octets per line. Option `a' outputs a\n\@@ -11284,18 +11383,18 @@ void help(std::ostream& out) arguments are never ignored. This option may be repeated.\n\ --include=GLOB\n\ Search only files whose name matches GLOB using wildcard matching,\n\- same as -g GLOB. GLOB can use **, *, ?, and [...] as wildcards,\n\- and \\ to quote a wildcard or backslash character literally. When\n\- GLOB contains a `/', full pathnames are matched. Otherwise\n\- basenames are matched. When GLOB ends with a `/', directories are\n\- included as if --include-dir is specified. Otherwise files are\n\- included. Note that --exclude patterns take priority over\n\- --include patterns. GLOB should be quoted to prevent shell\n\- globbing. This option may be repeated.\n\+ same as -g GLOB. GLOB can use **, *, ?, and [...] as wildcards and\n\+ \\ to quote a wildcard or backslash character literally. When GLOB\n\+ contains a `/', full pathnames are matched. Otherwise basenames\n\+ are matched. When GLOB ends with a `/', directories are included\n\+ as if --include-dir is specified. Otherwise files are included.\n\+ Note that --exclude patterns take priority over --include patterns.\n\+ GLOB should be quoted to prevent shell globbing. This option may\n\+ be repeated.\n\ --include-dir=GLOB\n\ Only directories whose name matches GLOB are included in recursive\n\ searches, same as -g GLOB/. GLOB can use **, *, ?, and [...] as\n\- wildcards, and \\ to quote a wildcard or backslash character\n\+ wildcards and \\ to quote a wildcard or backslash character\n\ literally. When GLOB contains a `/', full pathnames are matched.\n\ Otherwise basenames are matched. Note that --exclude-dir patterns\n\ take priority over --include-dir patterns. GLOB should be quoted\n\@@ -11303,7 +11402,7 @@ void help(std::ostream& out) --include-from=FILE\n\ Read the globs from FILE and search only files and directories\n\ whose name matches one or more globs. A glob can use **, *, ?, and\n\- [...] as wildcards, and \\ to quote a wildcard or backslash\n\+ [...] as wildcards and \\ to quote a wildcard or backslash\n\ character literally. When a glob contains a `/', full pathnames\n\ are matched. Otherwise basenames are matched. When a glob ends\n\ with a `/', directories are included as if --include-dir is\n\@@ -11395,13 +11494,13 @@ void help(std::ostream& out) each file processed.\n\ --no-group-separator\n\ Removes the group separator line from the output for context\n\- options -A, -B, and -C.\n\+ options -A, -B and -C.\n\ --not [-e] PATTERN\n\ Specifies that PATTERN should not match. Note that -e A --not -e B\n\ matches lines with `A' or lines without a `B'. To match lines with\n\ `A' that have no `B', specify -e A --andnot -e B. Option --stats\n\ displays the search patterns applied. See also options --and,\n\- --andnot, --bool, --files, and --lines.\n\+ --andnot, --bool, --files and --lines.\n\ -O EXTENSIONS, --file-extension=EXTENSIONS\n\ Search only files whose filename extensions match the specified\n\ comma-separated list of EXTENSIONS, same as --include='*.ext' for\n\@@ -11414,7 +11513,7 @@ void help(std::ostream& out) the line numbers with option -n are displayed using `|' as the\n\ field separator for each additional line matched by the pattern.\n\ If -u is specified, ungroups multiple matches on the same line.\n\- This option cannot be combined with options -A, -B, -C, -v, and -y.\n\+ This option cannot be combined with options -A, -B, -C, -v and -y.\n\ --only-line-number\n\ The line number of the matching line in the file is output without\n\ displaying the match. The line number counter is reset for each\n\@@ -11481,6 +11580,12 @@ void help(std::ostream& out) Recursively read all files under each directory, following symbolic\n\ links only if they are on the command line. When -J1 is specified,\n\ files are searched in the same order as specified.\n\+ --replace=FORMAT\n\+ Replace matching patterns in the output by the specified FORMAT\n\+ with `%' fields. When option -P is used, FORMAT may include `%1'\n\+ to `%9', `%[NUM]#' and `%[NAME]#' to output group captures. A `%%'\n\+ outputs `%' and `%~' outputs a newline. See option --format,\n\+ `ugrep --help format' and `man ugrep' section FORMAT for details.\n\ -S, --dereference\n\ If -r is specified, all symbolic links are followed, like -R. The\n\ default is not to follow symbolic links.\n\@@ -11492,7 +11597,7 @@ void help(std::ostream& out) a `-', write the configuration to standard output.\n\ --separator[=SEP]\n\ Use SEP as field separator between file name, line number, column\n\- number, byte offset, and the matched line. The default is a colon\n\+ number, byte offset and the matched line. The default is a colon\n\ (`:').\n\ --sort[=KEY]\n\ Displays matching files in the order specified by KEY in recursive\n\@@ -11500,18 +11605,18 @@ void help(std::ostream& out) to sort by best match with option -Z (sort by best match requires\n\ two passes over the input files), `size' to sort by file size,\n\ `used' to sort by last access time, `changed' to sort by last\n\- modification time, and `created' to sort by creation time. Sorting\n\+ modification time and `created' to sort by creation time. Sorting\n\ is reversed with `rname', `rbest', `rsize', `rused', `rchanged', or\n\ `rcreated'. Archive contents are not sorted. Subdirectories are\n\ sorted and displayed after matching files. FILE arguments are\n\ searched in the same order as specified. Normally ugrep displays\n\ matches in no particular order to improve performance.\n\ --stats\n\- Output statistics on the number of files and directories searched,\n\+ Output statistics on the number of files and directories searched\n\ and the inclusion and exclusion constraints applied.\n\ -T, --initial-tab\n\ Add a tab space to separate the file name, line number, column\n\- number, and byte offset with the matched line.\n\+ number and byte offset with the matched line.\n\ -t TYPES, --file-type=TYPES\n\ Search only files associated with TYPES, a comma-separated list of\n\ file types. Each file type corresponds to a set of filename\n\@@ -11540,7 +11645,7 @@ void help(std::ostream& out) Output the matched line again for each additional pattern match,\n\ using `+' as the field separator.\n\ -V, --version\n\- Display version information and exit.\n\+ Display version with linked libraries and exit.\n\ -v, --invert-match\n\ Selected lines are those not matching any of the specified\n\ patterns.\n\@@ -11552,9 +11657,9 @@ void help(std::ostream& out) -w, --word-regexp\n\ The PATTERN is searched for as a word, such that the matching text\n\ is preceded by a non-word character and is followed by a non-word\n\- character. Word characters are letters, digits, and the\n\+ character. Word characters are letters, digits and the\n\ underscore. With option -P, word characters are Unicode letters,\n\- digits, and underscore. This option has no effect if -x is also\n\+ digits and underscore. This option has no effect if -x is also\n\ specified. If a PATTERN is specified, or -e PATTERN or -N PATTERN,\n\ then this option has no effect on -f FILE patterns to allow -f FILE\n\ patterns to narrow or widen the scope of the PATTERN search.\n\@@ -11577,7 +11682,7 @@ void help(std::ostream& out) not only lines containing the character `x'.\n\ -y, --any-line\n\ Any line is output (passthru). Non-matching lines are output as\n\- context with a `-' separator. See also options -A, -B, and -C.\n\+ context with a `-' separator. See also options -A, -B and -C.\n\ -Z[[+-~]MAX], --fuzzy[=[+-~]MAX]\n\ Fuzzy mode: report approximate pattern matches within MAX errors.\n\ By default, MAX is 1: one deletion, insertion or substitution is\n\@@ -11623,14 +11728,14 @@ void help(std::ostream& out) "\ --zmax=NUM\n\ When used with option -z (--decompress), searches the contents of\n\- compressed files and archives stored within archives for up to NUM\n\- recursive levels deep. The default --zmax=1 only permits searching\n\+ compressed files and archives stored within archives by up to NUM\n\+ recursive expansions. The default --zmax=1 only permits searching\n\ uncompressed files stored in cpio, pax, tar and zip archives;\n\ compressed files and archives are detected as binary files and are\n\ effectively ignored. Specify --zmax=2 to search compressed files\n\ and archives stored in cpio, pax, tar and zip archives. NUM may\n\ range from 1 to 99 for up to 99 decompression and de-archiving\n\- steps. Larger NUM values gradually degrade performance.\n"+ steps. Increasing NUM values gradually degrades performance.\n" #ifndef WITH_DECOMPRESSION_THREAD "\ This option is not available in this build configuration of ugrep.\n"@@ -11655,82 +11760,223 @@ void help(std::ostream& out) // display helpful information for WHAT, if specified, and exit void help(const char *what) {- if (what == NULL)+ if (what != NULL && *what == '=')+ ++what;++ if (what != NULL && strncmp(what, "--no", 4) == 0)+ what += 4;++ if (what == NULL || *what == '\0') { help(std::cout); } else {- if (*what == '=')- ++what;+ std::stringstream text;+ help(text);+ const std::string& str = text.str(); - if (strncmp(what, "--no", 4) == 0)- what += 4;+ int found = 0; - if (*what == '\0')+ for (int pass = 0; pass < 2; ++pass) {- help(std::cout);- }- else- {- std::stringstream text;- help(text);- const std::string& str = text.str();-- int found = 0;+ size_t pos = 0; - for (int pass = 0; pass < 2; ++pass)+ while (true) {- size_t pos = 0;-- while (true)- {- size_t end = str.find("\n -", pos + 1);+ size_t end = str.find("\n -", pos + 1); - if (end == std::string::npos)- end = str.find("\n\n", pos + 1);+ if (end == std::string::npos)+ end = str.find("\n\n", pos + 1); - if (end == std::string::npos)- break;+ if (end == std::string::npos)+ break; - size_t nl = str.find('\n', pos + 1);+ size_t nl = str.find('\n', pos + 1); - // roughly find a case-independent match of WHAT- for (size_t i = pos + 5; i < (pass == 0 ? nl : end); ++i)- {- size_t j = 0;+ // roughly find a case-independent match of WHAT+ for (size_t i = pos + 5; i < (pass == 0 ? nl : end); ++i)+ {+ size_t j = 0; - for (j = 0; what[j] != '\0'; ++j)- if (((what[j] ^ str.at(i + j)) & ~0x20) != 0)- break;+ for (j = 0; what[j] != '\0'; ++j)+ if (((what[j] ^ str.at(i + j)) & ~0x20) != 0)+ break; - if (what[j] == '\0')+ if (what[j] == '\0')+ {+ if (pass == 0 ? i < nl: i > nl) {- if (pass == 0 ? i < nl: i > nl)- {- if (found == 0 && pass == 0)- std::cout << "\nOptions and arguments:\n";- else if (found == 1 && pass == 1)- std::cout << "\n\nOther options:\n";- else if (found == 0)- std::cout << "\nNo matching option, other relevant options:\n";-- std::cout << str.substr(pos, end - pos);- found = pass + 1;- }- break;+ if (found == 0 && pass == 0)+ std::cout << "\nOptions and arguments:\n";+ else if (found == 1 && pass == 1)+ std::cout << "\n\nOther options:\n";+ else if (found == 0)+ std::cout << "\nNo matching option, other relevant options:\n";++ std::cout << str.substr(pos, end - pos);+ found = pass + 1; }+ break; }-- pos = end; }++ pos = end; }+ }++ if (found == 0)+ std::cout << "ugrep --help: nothing appropriate for " << what; - if (found == 0)- std::cout << "ugrep --help: nothing appropriate for " << what;+ std::cout << "\n\n"; - std::cout << "\n\n";+ if (strcmp(what, "format") == 0)+ {+ std::cout <<+"FORMAT fields for --format and --replace:\n\+\n\+ field output field output\n\+ ---------- -------------------------- ---------- --------------------------\n\+ %a basename %% %\n\+ %b byte offset %~ newline\n\+ %B %[...]B ... + byte offset, if -b %[...]< ..., if %m = 1\n\+ %c matching pattern, as C++ %[...]> ..., if %m > 1\n\+ %C matching line, as C++ %, same as %[,]>\n\+ %d byte size %: same as %[:]>\n\+ %e end offset %; same as %[;]>\n\+ %f pathname %| same as %[|]>\n\+ %F %[...]F ... + pathname, if -H %[...]$ assign ... to separator\n\+ %h pathname --------------------------------------\n\+ %H %[...]H ... + \"pathname\", if -H\n\+ %j matching pattern, as JSON Fields that require -P for captures:\n\+ %J matching line, as JSON\n\+ %k line number field output\n\+ %K %[...]K ... + column number, if -k ---------- --------------------------\n\+ %m match number %1 %2...%9 group capture\n\+ %n line number %[n]# nth group capture\n\+ %N %[...]N ... + line number, if -n %[n]b nth capture byte offset\n\+ %o matching pattern %[n]d nth capture byte size\n\+ %O matching line %[n]e nth capture end offset\n\+ %p path %[name]# named group capture\n\+ %q quoted matching pattern %[name]b named capture byte offset\n\+ %Q quoted matching line %[name]d named capture byte size\n\+ %s separator %[name]e named capture end offset\n\+ %S %[...]S ... + separator, if %m > 1 %[n|...]# capture n,... that matched\n\+ %t tab %[n|...]b cpature n,... byte offset\n\+ %T %[...]T ... + tab, if -T %[n|...]d cpature n,... byte size\n\+ %u unique lines, unless -u %[n|...]e cpature n,... end offset\n\+ %v matching pattern, as CSV %[t|...]g text t indexed by capture\n\+ %V matching line, as CSV %[t|...]G all t indexed by captures\n\+ %w match width in wide chars --------------------------------------\n\+ %x matching pattern, as XML\n\+ %X matching line, as XML\n\+ %z path in archive\n\+ %Z edit distance cost, if -Z\n\+ --------------------------------------\n\+\n\+";+ }+ else if (strcmp(what, "regex") == 0)+ {+ std::cout <<+"Extended regular expression syntax overview (excludes some advanced patterns):\n\+\n\+ pattern matches pattern special characters\n\+ ---------- -------------------------- ---------- --------------------------\n\+ . any char except newline \\f form feed\n\+ a the character a \\n newline\n\+ ab the string ab \\r carriage return\n\+ a|b a or b \\R any Unicode newline\n\+ a* zero or more a's \\t tab\n\+ a+ one or more a's \\v vertical tab\n\+ a? zero or one a \\0 NUL\n\+ a{3} 3 a's \\cZ control character Z\n\+ a{3,} 3 or more a's \\xHH hex character code HH\n\+ a{3,7} 3 to 7 a's \\x{HHHH} hex character code HHHH\n\+ a*? zero or more a's lazily \\u{HHHH} hex character code HHHH\n\+ a+? one or more a's lazily --------------------------------------\n\+ a?? zero or one a lazily\n\+ a{3}? 3 a's lazily pattern anchors and boundaries\n\+ a{3,}? 3 or more a's lazily ---------- --------------------------\n\+ a{3,7}? 3 to 7 a's lazily ^ begin of line anchor\n\+ -------------------------------------- $ end of line anchor\n\+ \\b word boundary (-P)\n\+ pattern character classes \\B non-word boundary (-P)\n\+ ---------- -------------------------- \\< start word boundary (-P)\n\+ [abc-e] one character a,b,c,d,e \\> end word boundary (-P)\n\+ [^abc-e] one char not a,b,c,d,e,\\n (?=...) lookahead (-P)\n\+ [[:name:]] one char in POSIX class: (?!...) negative lookahead (-P)\n\+ alnum a-z,A-Z,0-9 (?<=...) lookbehind (-P)\n\+ alpha a-z,A-Z (?<!...) negative lookbehind (-P)\n\+ ascii ASCII char \\x00-\\x7f --------------------------------------\n\+ blank space or tab\n\+ cntrl control characters pattern grouping\n\+ digit 0-9 ---------- --------------------------\n\+ graph visible characters (...) capturing group (-P)\n\+ lower a-z (...) non-capturing group\n\+ print visible chars and space (?:...) non-capturing group\n\+ punct punctuation characters (?<X>...) capturing, named X (-P)\n\+ space space,\\t,\\n,\\v,\\f,\\r \\1 matches group 1 (-P)\n\+ upper A-Z \\g{10} matches group 10 (-P)\n\+ word a-z,A-Z,0-9,_ \\g{X} matches group X (-P)\n\+ xdigit 0-9,a-f,A-F (?#...) comments ... are ignored\n\+ \\p{class} one char in class --------------------------------------\n\+ \\P{class} one non-char in class\n\+ \\d a digit pattern removes special meaning\n\+ \\D a non-digit ---------- --------------------------\n\+ \\h a space or tab \\. escapes . and other chars\n\+ \\H not a space or tab \\Q...\\E the literal string ...\n\+ \\s a whitespace except \\n --------------------------------------\n\+ \\S a non-whitespace\n\+ \\w a word character\n\+ \\W a non-word character\n\+ -------------------------------------- \n\+\n\+Option -P is required by certain regex patterns as indicated by (-P).\n\+\n\+Option -U disables full Unicode pattern matching. Non-POSIX Unicode character\n\+classes \\p{class} are disabled. Only ASCII, LATIN1 or binary regex patterns\n\+may be specified.\n\+\n\+Option --bool adds the following syntax and conventions to patterns, listed\n\+from the lowest to the highest level of precedence:\n\+\n\+ pattern operation pattern operation\n\+ ---------- -------------------------- ---------- --------------------------\n\+ space logical AND AND locical AND\n\+ | logical OR OR logical OR\n\+ - logical NOT NOT locical NOT\n\+ (...) grouping \"...\" the literal string ...\n\+ -------------------------------------- --------------------------------------\n\+\n\+"; }+ else if (strcmp(what, "globs") == 0)+ {+ std::cout <<+"Glob syntax and conventions:\n\+\n\+ * anything except a /\n\+ ? any one character except a /\n\+ [a-z] one character in the selected range of characters\n\+ [^a-z] one character not in the selected range of characters\n\+ [!a-z] one character not in the selected range of characters\n\+ / when used at the begin of a glob, matches working directory\n\+ **/ zero or more directories\n\+ /** when at the end of a glob, matches everything after the /\n\+ \\? a ? (or any character specified after the backslash)\n\+\n\+When a glob contains a /, the full pathname is matched. Otherwise, the\n\+basename of a file or directory is matched in recursive searches.\n\+\n\+When a glob begins with a /, files and directories are matched at the working\n\+directory, not recursively.\n\+\n\+When a glob ends with a /, directories are matched instead of files.\n\+\n\+";+ }+ } exit(EXIT_ERROR);
@@ -30,7 +30,7 @@ @file ugrep.hpp @brief a pattern search utility written in C++11 @author Robert van Engelen - engelen@genivia.com-@copyright (c) 2019-2021, Robert van Engelen, Genivia Inc. All rights reserved.+@copyright (c) 2019-2022, Robert van Engelen, Genivia Inc. All rights reserved. @copyright (c) BSD-3 License - see LICENSE.txt */ @@ -38,7 +38,7 @@ #define UGREP_HPP // ugrep version-#define UGREP_VERSION "3.5.0"+#define UGREP_VERSION "3.6.0" // disable mmap because mmap is almost always slower than the file reading speed improvements since 3.0.0 #define WITH_NO_MMAP
@@ -30,7 +30,7 @@ @file vkey.cpp @brief Virtual terminal keyboard input API - static, not thread safe @author Robert van Engelen - engelen@genivia.com-@copyright (c) 2019-2020, Robert van Engelen, Genivia Inc. All rights reserved.+@copyright (c) 2019-2022, Robert van Engelen, Genivia Inc. All rights reserved. @copyright (c) BSD-3 License - see LICENSE.txt */
@@ -30,7 +30,7 @@ @file vkey.hpp @brief Virtual terminal keyboard input API - static, not thread safe @author Robert van Engelen - engelen@genivia.com-@copyright (c) 2019-2020, Robert van Engelen, Genivia Inc. All rights reserved.+@copyright (c) 2019-2022, Robert van Engelen, Genivia Inc. All rights reserved. @copyright (c) BSD-3 License - see LICENSE.txt Supports Unix/Linux, MacOS, DOS, and Windows.
@@ -30,7 +30,7 @@ @file zstream.hpp @brief file decompression streams - zstreambuf extends std::streambuf @author Robert van Engelen - engelen@genivia.com-@copyright (c) 2019-2021, Robert van Engelen, Genivia Inc. All rights reserved.+@copyright (c) 2019-2022, Robert van Engelen, Genivia Inc. All rights reserved. @copyright (c) BSD-3 License - see LICENSE.txt */
@@ -105,6 +105,7 @@ for PAT in '' 'Hello' '\w+\s+\S+' '\S\n\S' 'nomatch' ; do $UG -U --tag "$PAT" $FILES > "out/$FN--tag.out" $UG -U --format-open='%m) %f:%~' --format=' %m) %n,%k %w-%d%~' --format-close='%~' "$PAT" $FILES > "out/$FN--format.out" $UG -U -v --format-open='%m) %f:%~' --format=' %m) %n,%k %w-%d%~' --format-close='%~' "$PAT" $FILES > "out/$FN-v--format.out"+ $UG -U --replace='(%m:%o)' "$PAT" $FILES > "out/$FN--replace.out" $UG -U -Iw "$PAT" $FILES > "out/$FN-Iw.out" $UG -U -Ix "$PAT" $FILES > "out/$FN-Ix.out" $UG -U -F "$PAT" $FILES > "out/$FN-F.out"
@@ -0,0 +1,14 @@+[1;35mHello.bat[m[1;36m:[m[m(1:#!/bin/sh)[m+[1;35mHello.bat[m[1;36m:[m[m(2:echo "Hello World!")[m+[mBinary file[1;35m Hello.class[m matches+[1;35mHello.java[m[1;36m:[m[m(1:import java.lang.*;)[m+[1;35mHello.java[m[1;36m:[m[m(2:)[m+[1;35mHello.java[m[1;36m:[m[m(3:public class Hello // prints a Hello World! greeting)[m+[1;35mHello.java[m[1;36m:[m[m(4:{ public static void main(String[] arg))[m+[1;35mHello.java[m[1;36m:[m[m(5: { System.out.println("Hello World!");)[m+[1;35mHello.java[m[1;36m:[m[m(6: })[m+[1;35mHello.java[m[1;36m:[m[m(7:})[m+[mBinary file[1;35m Hello.pdf[m matches+[1;35mHello.sh[m[1;36m:[m[m(1:#!/bin/sh)[m+[1;35mHello.sh[m[1;36m:[m[m(2:echo "Hello World!")[m+[1;35mHello.txt[m[1;36m:[m[m(1:Hello)[m
tests/out/Hello_Hello--replace.out
@@ -0,0 +1,7 @@+[1;35mHello.bat[m[1;36m:[mecho "[m[1;4;32m(1:Hello)[m World!"[m+[mBinary file[1;35m Hello.class[m matches+[1;35mHello.java[m[1;36m:[mpublic class [m[1;4;32m(1:Hello)[m // prints a [m[1;4;32m(1:Hello)[m World! greeting[m+[1;35mHello.java[m[1;36m:[m { System.out.println("[m[1;4;32m(2:Hello)[m World!");[m+[mBinary file[1;35m Hello.pdf[m matches+[1;35mHello.sh[m[1;36m:[mecho "[m[1;4;32m(1:Hello)[m World!"[m+[1;35mHello.txt[m[1;36m:[m[m[1;4;32m(1:Hello)[m
tests/out/Hello_SnS--replace.out
@@ -0,0 +1,10 @@+[1;35mHello.bat[m[1;36m:[m#!/bin/s[m[1;4;32m(1:h+e)[mcho "Hello World!"[m+[mBinary file[1;35m Hello.class[m matches+[1;35mHello.java[m[1;36m:[mpublic class Hello // prints a Hello World! greetin[m[1;4;32m(1:g+{)[m public static void main(String[] arg)[m+[1;35mHello.java[m[1;36m:[m [m[1;4;32m(2:}+})[m+[mBinary file[1;35m Hello.pdf[m matches+[1;35mHello.sh[m[1;36m:[m#!/bin/s[m[1;4;32m(1:h+e)[mcho "Hello World!"[m
tests/out/Hello_nomatch--replace.out
tests/out/Hello_wsS--replace.out
@@ -0,0 +1,8 @@+[1;35mHello.bat[m[1;36m:[m[m[1;4;32m(1:echo "Hello)[m World!"[m+[mBinary file[1;35m Hello.class[m matches+[1;35mHello.java[m[1;36m:[m[m[1;4;32m(1:import java.lang.*;)[m+[1;35mHello.java[m[1;36m:[m[m[1;4;32m(2:public class)[m [m[1;4;32m(2:Hello //)[m [m[1;4;32m(2:prints a)[m [m[1;4;32m(2:Hello World!)[m greeting[m+[1;35mHello.java[m[1;36m:[m{ [m[1;4;32m(3:public static)[m [m[1;4;32m(3:void main(String[])[m arg)[m+[1;35mHello.java[m[1;36m:[m { System.out.println("[m[1;4;32m(4:Hello World!");)[m+[mBinary file[1;35m Hello.pdf[m matches+[1;35mHello.sh[m[1;36m:[m[m[1;4;32m(1:echo "Hello)[m World!"[m
@@ -246,6 +246,10 @@ for PAT in '' 'Hello' '\w+\s+\S+' '\S\n\S' 'nomatch' ; do | $DIFF "out/$FN-v--format.out" \ || ERR "-v --format-open='%m) %f:%~' --format=' %m) %n,%k %w-%d%~' --format-close='%~' '$PAT' $FILES" printf .+ $UG -U --replace='(%m:%o)' "$PAT" $FILES \+ | $DIFF "out/$FN--replace.out" \+ || ERR "--replace='(%m:%o)' '$PAT' $FILES"+ printf . $UG -U -Iw "$PAT" $FILES \ | $DIFF "out/$FN-Iw.out" \ || ERR "-Iw '$PAT' $FILES"
@@ -150,7 +150,7 @@ Open vs\ugrep\ugrep.sln in Visual Studio. Upgrade the version if prompted. Edit Visual Studio project properties for Release x86 to make sure these match the following: -Configuration Properties+Project Configuration Properties General: Use of MFC: Use Standard Windows Libraries C/C++@@ -172,6 +172,18 @@ Configuration Properties Input and Output: Additional Manifest Files: $(ProjectDir)\..\manifest.xml +Right-click on the matcher_avx2.cpp file to set the C++ preprocessor properties specifically for this file by adding COMPILE_AVX2 to the list of preprocessor definitions, and enable AVX2:++ C/C+++ Preprocessor: + Preprocessor Definitions: WIN32;NDEBUG;_CONSOLE;WITH_NO_INDENT;HAVE_AVX2;COMPILE_AVX2;HAVE_PCRE2;PCRE2_STATIC;HAVE_LIBZ;HAVE_LIBBZ2;HAVE_LIBLZMA;HAVE_LIBLZ4;HAVE_LIBZSTD;WITH_COLOR;ZLIB_WINAPI;NO_GZCOMPRESS;LZMA_API_STATIC;_CRT_NONSTDC_NO_DEPRECATE;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_WARNINGS+ Code Generation:+ Enable Enhanced Instruction Set: Advanced Vector Extensions 2 (/arch:AVX2) ++If the general C++ preprocessor properties and the C++ preprocessor properties for matcher_avx2.cpp are the same, then linkage fails with duplicate definition errors.++Remove matcher_avx512bw.cpp from the list of source files.+ Then build ugrep in Visual Studio. @@ -258,7 +270,7 @@ Open vs\ugrep\ugrep.sln in Visual Studio. Upgrade the version if prompted. Edit Visual Studio project properties for Release x86 to make sure these match the following: -Configuration Properties+Project Configuration Properties General: Use of MFC: Use Standard Windows Libraries C/C++@@ -280,4 +292,16 @@ Configuration Properties Input and Output: Additional Manifest Files: $(ProjectDir)\..\manifest.xml +Right-click on the matcher_avx2.cpp file to set the C++ preprocessor properties specifically for this file by adding COMPILE_AVX2 to the list of preprocessor definitions, and enable AVX2:++ C/C+++ Preprocessor: + Preprocessor Definitions: WIN32;NDEBUG;_CONSOLE;WITH_NO_INDENT;COMPILE_AVX2;HAVE_AVX2;HAVE_BOOST_REGEX;HAVE_LIBZ;HAVE_LIBBZ2;HAVE_LIBLZMA;HAVE_LIBLZ4;HAVE_LIBZSTD;WITH_COLOR;ZLIB_WINAPI;NO_GZCOMPRESS;LZMA_API_STATIC;_CRT_NONSTDC_NO_DEPRECATE;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_WARNINGS+ Code Generation:+ Enable Enhanced Instruction Set: Advanced Vector Extensions 2 (/arch:AVX2) ++If the general C++ preprocessor properties and the C++ preprocessor properties for matcher_avx2.cpp are the same, then linkage fails with duplicate definition errors.++Remove matcher_avx512bw.cpp from the list of source files.+ Then build ugrep in Visual Studio.
@@ -44,8 +44,16 @@ <ClCompile Include="lib\language_scripts.cpp" /> <ClCompile Include="lib\letter_scripts.cpp" /> <ClCompile Include="lib\matcher.cpp" />+ <ClCompile Include="lib\matcher_avx2.cpp">+ <PreprocessorDefinitions Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">WIN32;NDEBUG;_CONSOLE;WITH_NO_INDENT;COMPILE_AVX2;HAVE_AVX2;HAVE_PCRE2;PCRE2_STATIC;HAVE_LIBZ;HAVE_LIBBZ2;HAVE_LIBLZMA;HAVE_LIBLZ4;HAVE_LIBZSTD;WITH_COLOR;WITH_NO_HIDDEN;ZLIB_WINAPI;NO_GZCOMPRESS;LZMA_API_STATIC;_CRT_NONSTDC_NO_DEPRECATE;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_WARNINGS;%(PreprocessorDefinitions)</PreprocessorDefinitions>+ <PreprocessorDefinitions Condition="'$(Configuration)|$(Platform)'=='Release|x64'">NDEBUG;_CONSOLE;WITH_NO_INDENT;COMPILE_AVX2;HAVE_AVX2;HAVE_PCRE2;PCRE2_STATIC;HAVE_LIBZ;HAVE_LIBBZ2;HAVE_LIBLZMA;HAVE_LIBLZ4;HAVE_LIBZSTD;WITH_COLOR;WITH_NO_HIDDEN;ZLIB_WINAPI;NO_GZCOMPRESS;LZMA_API_STATIC;_CRT_NONSTDC_NO_DEPRECATE;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_WARNINGS;%(PreprocessorDefinitions)</PreprocessorDefinitions>+ <EnableEnhancedInstructionSet Condition="'$(Configuration)|$(Platform)'=='Release|x64'">AdvancedVectorExtensions2</EnableEnhancedInstructionSet>+ <EnableEnhancedInstructionSet Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">AdvancedVectorExtensions2</EnableEnhancedInstructionSet>+ </ClCompile> <ClCompile Include="lib\pattern.cpp" /> <ClCompile Include="lib\posix.cpp" />+ <ClCompile Include="lib\simd_avx2.cpp" />+ <ClCompile Include="lib\simd_avx512bw.cpp" /> <ClCompile Include="lib\unicode.cpp" /> <ClCompile Include="lib\utf8.cpp" /> <ClCompile Include="src\cnf.cpp" />@@ -54,10 +62,7 @@ <ClCompile Include="src\query.cpp" /> <ClCompile Include="src\screen.cpp" /> <ClCompile Include="src\stats.cpp" />- <ClCompile Include="src\ugrep.cpp">- <PreprocessorDefinitions Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">WIN32;NDEBUG;_CONSOLE;WITH_NO_INDENT;HAVE_AVX;HAVE_PCRE2;PCRE2_STATIC;HAVE_LIBZ;HAVE_LIBBZ2;HAVE_LIBLZMA;WITH_COLOR;ZLIB_WINAPI;NO_GZCOMPRESS;LZMA_API_STATIC;_CRT_NONSTDC_NO_DEPRECATE;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_WARNINGS;%(PreprocessorDefinitions)</PreprocessorDefinitions>- <PreprocessorDefinitions Condition="'$(Configuration)|$(Platform)'=='Release|x64'">NDEBUG;_CONSOLE;WITH_NO_INDENT;HAVE_AVX;HAVE_PCRE2;PCRE2_STATIC;HAVE_LIBZ;HAVE_LIBBZ2;HAVE_LIBLZMA;WITH_COLOR;ZLIB_WINAPI;NO_GZCOMPRESS;LZMA_API_STATIC;_CRT_NONSTDC_NO_DEPRECATE;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_WARNINGS;%(PreprocessorDefinitions)</PreprocessorDefinitions>- </ClCompile>+ <ClCompile Include="src\ugrep.cpp" /> <ClCompile Include="src\vkey.cpp" /> <ClCompile Include="src\zopen.c" /> </ItemGroup>@@ -80,6 +85,7 @@ <ClInclude Include="include\reflex\posix.h" /> <ClInclude Include="include\reflex\ranges.h" /> <ClInclude Include="include\reflex\setop.h" />+ <ClInclude Include="include\reflex\simd.h" /> <ClInclude Include="include\reflex\stdmatcher.h" /> <ClInclude Include="include\reflex\timer.h" /> <ClInclude Include="include\reflex\traits.h" />@@ -209,9 +215,10 @@ <PreprocessorDefinitions>WIN32;NDEBUG;_CONSOLE;WITH_NO_INDENT;HAVE_AVX2;HAVE_PCRE2;PCRE2_STATIC;HAVE_LIBZ;HAVE_LIBBZ2;HAVE_LIBLZMA;HAVE_LIBLZ4;HAVE_LIBZSTD;WITH_COLOR;WITH_NO_HIDDEN;ZLIB_WINAPI;NO_GZCOMPRESS;LZMA_API_STATIC;_CRT_NONSTDC_NO_DEPRECATE;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_WARNINGS;%(PreprocessorDefinitions)</PreprocessorDefinitions> <SDLCheck>true</SDLCheck> <RuntimeLibrary>MultiThreaded</RuntimeLibrary>- <AdditionalIncludeDirectories>$(ProjectDir)\include;$(ProjectDir)\..\pcre2-10.34\src;$(ProjectDir)\..\zlib-1.2.11;$(ProjectDir)\..\bzip2-1.0.5;$(ProjectDir)\..\api;$(ProjectDir)\..\lz4-dev\lib;$(ProjectDir)\..\zstd-dev\lib;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>+ <AdditionalIncludeDirectories>$(ProjectDir)\include;$(ProjectDir)\..\pcre2-10.37\src;$(ProjectDir)\..\zlib-1.2.11;$(ProjectDir)\..\bzip2-1.0.5;$(ProjectDir)\..\api;$(ProjectDir)\..\lz4-dev\lib;$(ProjectDir)\..\zstd-dev\lib;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories> <DisableSpecificWarnings> </DisableSpecificWarnings>+ <FavorSizeOrSpeed>Speed</FavorSizeOrSpeed> </ClCompile> <Link> <SubSystem>Console</SubSystem>@@ -221,6 +228,9 @@ <AdditionalDependencies>pcre2-x32.lib;liblzma-x32.lib;libzstd_static.lib;%(AdditionalDependencies)</AdditionalDependencies> <AdditionalLibraryDirectories>$(ProjectDir)\..;$(ProjectDir)\..\zstd-dev\build\VS2010\bin\Win32_Release;%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories> </Link>+ <Manifest>+ <AdditionalManifestFiles>$(ProjectDir)\..\manifest.xml %(AdditionalManifestFiles)</AdditionalManifestFiles>+ </Manifest> </ItemDefinitionGroup> <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'"> <ClCompile>@@ -232,7 +242,8 @@ <PreprocessorDefinitions>NDEBUG;_CONSOLE;WITH_NO_INDENT;HAVE_AVX2;HAVE_PCRE2;PCRE2_STATIC;HAVE_LIBZ;HAVE_LIBBZ2;HAVE_LIBLZMA;HAVE_LIBLZ4;HAVE_LIBZSTD;WITH_COLOR;WITH_NO_HIDDEN;ZLIB_WINAPI;NO_GZCOMPRESS;LZMA_API_STATIC;_CRT_NONSTDC_NO_DEPRECATE;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_WARNINGS;%(PreprocessorDefinitions)</PreprocessorDefinitions> <SDLCheck>true</SDLCheck> <RuntimeLibrary>MultiThreaded</RuntimeLibrary>- <AdditionalIncludeDirectories>$(ProjectDir)\include;$(ProjectDir)\..\pcre2-10.34\src;$(ProjectDir)\..\zlib-1.2.11;$(ProjectDir)\..\bzip2-1.0.5;$(ProjectDir)\..\api;$(ProjectDir)\..\lz4-dev\lib;$(ProjectDir)\..\zstd-dev\lib</AdditionalIncludeDirectories>+ <AdditionalIncludeDirectories>$(ProjectDir)\include;$(ProjectDir)\..\pcre2-10.37\src;$(ProjectDir)\..\zlib-1.2.11;$(ProjectDir)\..\bzip2-1.0.5;$(ProjectDir)\..\api;$(ProjectDir)\..\lz4-dev\lib;$(ProjectDir)\..\zstd-dev\lib</AdditionalIncludeDirectories>+ <FavorSizeOrSpeed>Speed</FavorSizeOrSpeed> </ClCompile> <Link> <SubSystem>Console</SubSystem>@@ -243,6 +254,9 @@ <AdditionalLibraryDirectories>$(ProjectDir)\..;$(ProjectDir)\..\zstd-dev\build\VS2010\bin\x64_Release;%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories> <IgnoreAllDefaultLibraries>false</IgnoreAllDefaultLibraries> </Link>+ <Manifest>+ <AdditionalManifestFiles>$(ProjectDir)\..\manifest.xml %(AdditionalManifestFiles)</AdditionalManifestFiles>+ </Manifest> </ItemDefinitionGroup> <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" /> <ImportGroup Label="ExtensionTargets">
vs/ugrep/ugrep/ugrep.vcxproj.filters
@@ -129,6 +129,15 @@ <ClCompile Include="src\cnf.cpp"> <Filter>Source Files</Filter> </ClCompile>+ <ClCompile Include="lib\simd_avx512bw.cpp">+ <Filter>Source Files</Filter>+ </ClCompile>+ <ClCompile Include="lib\simd_avx2.cpp">+ <Filter>Source Files</Filter>+ </ClCompile>+ <ClCompile Include="lib\matcher_avx2.cpp">+ <Filter>Source Files</Filter>+ </ClCompile> </ItemGroup> <ItemGroup> <ClInclude Include="src\glob.hpp">@@ -242,5 +251,8 @@ <ClInclude Include="src\flag.hpp"> <Filter>Header Files</Filter> </ClInclude>+ <ClInclude Include="include\reflex\simd.h">+ <Filter>Header Files</Filter>+ </ClInclude> </ItemGroup> </Project>