250422.1 · a914ea4f0e1ba850e6a2b3dd4a70ec757ecbaf64
released 7.4
implement important improvements requested by users; refactored some parts of the source code for clarity
reading declarations
@@ -2053,17 +2053,18 @@ comments: .tbz2, .tb2, .tz2, .tlz, .txz, .tzst) are searched and matching pathnames of files in archives are output in braces. When used with option --zmax=NUM, searches the contents of compressed files- and archives stored within archives up to NUM levels. If -g, -O,- -M, or -t is specified, searches files stored in archives whose- filenames match globs, match filename extensions, match file- signature magic bytes, or match file types, respectively.+ and archives stored within archives up to NUM levels. When -g, -O,+ -M, or -t is specified, searches archives for files that match the+ specified globs, filename extensions, file signature magic bytes,+ or file types, respectively; a side-effect of these options is that+ the compressed files and archives searched are only those with+ filename extensions that match known compression and archive types. Supported compression formats: gzip (.gz), compress (.Z), zip, 7z,- bzip2 (requires suffix .bz, .bz2, .bzip2, .tbz, .tbz2, .tb2, .tz2),- lzma and xz (requires suffix .lzma, .tlz, .xz, .txz),+ bzip2 (.bz, .bz2, .bzip2, .tbz, .tbz2, .tb2, .tz2),+ xz (.xz, .txz) and lzma (requires suffix .lzma, .tlz),+ zstd (.zst, .zstd, .tzst), lz4 (requires suffix .lz4),- zstd (requires suffix .zst, .zstd, .tzst),- brotli (requires suffix .br),- bzip3 (requires suffix .bz3).+ brotli (requires suffix .br). --zmax=NUM When used with option -z (--decompress), searches the contents of compressed files and archives stored within archives by up to NUM@@ -2121,32 +2122,30 @@ When option `-z` is used with options `-g`, `-O`, `-M`, or `-t` to search cpio, jar, pax, tar, zip and 7z archives, archived files that match the filename selection criteria are searched only. -The gzip, compress, and zip formats are automatically detected, which is useful-when reading gzip-compressed data from standard input, e.g. input redirected-from a pipe. Other compression formats require a filename suffix: `.bz`,-`.bz2`, or `.bzip2` for bzip2, `.lzma` for lzma, `.xz` for xz, `.lz4` for lz4,-`.zst` or `.zstd` for zstd, `.br` for brotli and `.bz3` for bzip3. Also the-compressed tar archive shorthands `.taz`, `.tgz` and `.tpz` for gzip, `.tbz`,-`.tbz2`, `.tb2`, and `.tz2` for bzip2, `.tlz` for lzma, `.txz` for xz, and-`.tzst` for zstd are recognized. To search these formats with ugrep from-standard input, use option `--label='stdin.bz2'` for bzip2,-`--label='stdin.lzma'` for lzma, `--label='stdin.xz'` for xz,-`--label='stdin.lz4` for lz4 and `--label='stdin.zst` for zstd and so on. The-name `stdin` is arbitrary and may be omitted:--format | filename suffix | tar/pax archive short suffix | suffix required? | ugrep from stdin | library |+The gzip, compress, zip, bzip2, xz, and zstd formats are automatically+detected, which is useful when reading compressed data from standard input,+e.g. input redirected from a pipe. Other compression formats require a+filename suffix: `.lzma` for lzma, `.lz4` for lz4, `.br` for brotli and `.bz3`+for bzip3. Also the compressed tar archive shorthands `.taz`, `.tgz` and+`.tpz` for gzip, `.tbz`, `.tbz2`, `.tb2`, and `.tz2` for bzip2, `.tlz` for+lzma, `.txz` for xz, and `.tzst` for zstd are recognized. To search these+formats with ugrep from standard input, use option `--label='stdin.bz2'` for+bzip2, `--label='stdin.lzma'` for lzma, `--label='stdin.lz4` for lz4 and so on.+The name `stdin` is arbitrary and may be omitted:++format | filename suffix | tar/pax archive short suffix | suffix required? | detect on stdin | library | --------- | ----------------------- | ------------------------------- | ---------------- | ---------------- | ------------ | gzip | `.gz` | `.taz`, `.tgz`, `.tpz` | no | automatic | libz | compress | `.Z` | `.taZ`, `.tZ` | no | automatic | *built-in* | zip | `.zip`, `.zipx`, `.ZIP` | | no | automatic | libz |+bzip2 | `.bz`, `.bz2`, `.bzip2` | `.tb2`, `.tbz`, `.tbz2`, `.tz2` | no | automatic | libbz2 |+xz | `.xz` | `.txz` | no | automatic | liblzma |+zstd | `.zst`, `.zstd` | `.tzst` | no | automatic | libzstd | 7zip | `.7z` | | yes | `--label=.7z` | *built-in* |-bzip2 | `.bz`, `.bz2`, `.bzip2` | `.tb2`, `.tbz`, `.tbz2`, `.tz2` | yes | `--label=.bz2` | libbz2 |-lzma | `.lzma` | `.tlz` | yes | `--label=.lzma` | liblzma |-xz | `.xz` | `.txz` | yes | `--label=.xz` | liblzma |-lz4 | `.lz4` | | yes | `--label=.lz4` | liblz4 |-zstd | `.zst`, `.zstd` | `.tzst` | yes | `--label=.zst` | libzstd | brotli | `.br` | | yes | `--label=.br` | libbrotlidec | bzip3 | `.bz3` | | yes | `--label=.bz3` | libbzip3 |+lzma | `.lzma` | `.tlz` | yes | `--label=.lzma` | liblzma |+lz4 | `.lz4` | | yes | `--label=.lz4` | liblz4 | The gzip, bzip2, xz, lz4 and zstd formats support concatenated compressed files. Concatenated compressed files are searched as one single file.@@ -2189,12 +2188,12 @@ To recursively search C++ files including compressed files for the word ug -z -r -tc++ -Fw my_function -f cpp/zap_comments -To search bzip2, lzma, xz, lz4 and zstd compressed data on standard input,+To search lzma, lz4, brotli, and bzip3 compressed data on standard input, option `--label` may be used to specify the extension corresponding to the compression format to force decompression when the bzip2 extension is not available to ugrep, for example: - cat myfile.bz2 | ugrep -z --label='stdin.bz2' 'xyz'+ cat myfile.lz4 | ugrep -z --label='stdin.lz4' 'xyz' To search file `main.cpp` in `project.zip` for `TODO` and `FIXME` lines: @@ -2824,6 +2823,10 @@ implicit: previously-specified exclusions by including matching files. Lines starting with a `#' and empty lines in FILE are ignored. When FILE is a `-', standard input is read. This option may be repeated.+ --from=FILE+ Read additional pathnames of files to search from FILE. When FILE+ is a `-', standard input is read. This option is useful with `find+ ... -print | ugrep --from=- ...' to search specific files. --ignore-files[=FILE] Ignore files and directories matching the globs in each FILE that is encountered in recursive searches. The default FILE is@@ -3954,10 +3957,10 @@ in markdown: ugrep, ug -- file pattern searcher SYNOPSIS- ugrep [OPTIONS] [-i] [-Q|PATTERN] [-e PATTERN] [-N PATTERN] [-f FILE]- [-F|-G|-P|-Z] [-U] [-m [MIN,][MAX]] [--bool [--files|--lines]]- [-r|-R|-1|...|-9|-10|...] [-t TYPES] [-g GLOBS] [--sort[=KEY]]- [-l|-c] [-o] [-n] [-k] [-b] [-A NUM] [-B NUM] [-C NUM] [-y]+ ugrep [OPTIONS] [-i|-j] [-Q|PATTERN] [-e PATTERN] [-N PATTERN] [-f FILE]+ [-E|-F|-G|-P|-Z] [-U] [-m [MIN,][MAX]] [--bool [--files|--lines]]+ [-r|-R|-1|...|-9|-10|-11|...] [-t TYPES] [-g GLOBS] [--sort[=KEY]]+ [-q|-l|-c|-o] [-v] [-n] [-k] [-b] [-A NUM] [-B NUM] [-C NUM] [-y] [--color[=WHEN]|--colour[=WHEN]] [--pretty] [--pager[=COMMAND]] [--hexdump|--csv|--json|--xml] [-I] [-z] [--zmax=NUM] [FILE ...] @@ -3969,10 +3972,11 @@ in markdown: pattern matches every line. Each input line that matches at least one of the patterns is written to the standard output. - The ug command is intended for interactive searching, using a .ugrep- configuration file located in the working directory or home directory,- see CONFIGURATION. ug is equivalent to ugrep --config --pretty --sort to- load a .ugrep file, enhance the terminal output, and sort files by name.+ The ug command is intended for interactive searching, using a `.ugrep'+ configuration file located in the working directory or, if not found, in+ the home directory, see CONFIGURATION. ug is equivalent to ugrep+ --config --pretty --sort to load a configuration file, enhance the+ terminal output, and to sort files by name. The ugrep+ and ug+ commands are the same as the ugrep and ug commands, but also use filters to search pdfs, documents, e-books, and image@@ -3990,9 +3994,9 @@ in markdown: --files) to match files. Options -F (--fixed-strings), -G (--basic- regexp) and -P (--perl-regexp) specify other pattern syntaxes. - Option -i (--ignore-case) ignores case in ASCII patterns. When combined- with option -P, ignores case in Unicode patterns. Option -j (--smart-- case) enables -i only if the search patterns are specified in lower case.+ Option -i (--ignore-case) ignores letter case in patterns. Option -j+ (--smart-case) enables -i only if the search patterns are specified in+ lower case. Fuzzy (approximate) search is specified with option -Z (--fuzzy) with an optional argument to control character insertions, deletions, and/or@@ -4046,21 +4050,22 @@ in markdown: When used with option --zmax=NUM, searches the contents of compressed files and archives stored within archives up to NUM levels. + Output to a terminal for viewing is enhanced with --pretty, which is+ enabled by default with the ug command.+ A query terminal user interface (TUI) is opened with -Q (--query) to interactively specify search patterns and view search results. A PATTERN argument requires -e PATTERN to start the query TUI with the specified pattern. - Output to a terminal for viewing is enhanced with --pretty, which is- enabled by default with the ug command.-- A terminal output pager is enabled with --pager.+ A terminal output pager is enabled with --pager that accepts an optional+ COMMAND to view the output. Customized output is produced with option --format or --replace. Try ug- --help format for help with custom formatting of the output. Predefined- formats include CSV with option --csv, JSON with option --json, and XML- with option --xml. Hexdumps are output with option -X (--hex) or with- option --hexdump to customize hexdumps. See also section FORMAT.+ --help format for help with custom formatting of the output. See also+ section FORMAT. Predefined formats include CSV with option --csv, JSON+ with option --json, and XML with option --xml. Hexdumps are output with+ option -X (--hex) or with option --hexdump to customize hexdumps. A `--' signals the end of options; the rest of the parameters are FILE arguments, allowing filenames to begin with a `-' character.@@ -4209,8 +4214,8 @@ in markdown: directory. The options specified in the configuration FILE are parsed first, followed by the remaining options specified on the command line. The ug command automatically loads a `.ugrep'- configuration file, unless --config=FILE or --no-config is- specified.+ configuration file when present, unless --config=FILE or+ --no-config is specified. --no-config Do not automatically load the default .ugrep configuration file.@@ -4261,7 +4266,7 @@ in markdown: Specify a PATTERN to search the input. An input line is selected if it matches any of the specified patterns. This option is useful when multiple -e options are used to specify multiple- patterns, or when a pattern begins with a dash (`-'), or to+ patterns, or when a pattern starts with a dash (`-'), or to specify a pattern after option -f or after the FILE arguments. --encoding=ENCODING@@ -4291,7 +4296,7 @@ in markdown: --exclude-dir=GLOB Exclude directories whose name matches GLOB from recursive- searches, same as -g ^GLOB/. GLOB can use **, *, ?, and [...] as+ searches, same as -g ^GLOB/. GLOB may use **, *, ?, and [...] as wildcards and \ to quote a wildcard or backslash character literally. When GLOB contains a `/', full pathnames are matched. Otherwise basenames are matched. Note that --exclude-dir patterns@@ -4300,7 +4305,7 @@ in markdown: --exclude-from=FILE Read the globs from FILE and skip files and directories whose name- matches one or more globs. A glob can use **, *, ?, and [...] as+ matches one or more globs. A glob may use **, *, ?, and [...] as wildcards and \ to quote a wildcard or backslash character literally. When a glob contains a `/', full pathnames are matched. Otherwise basenames are matched. When a glob ends with@@ -4372,21 +4377,28 @@ in markdown: --free-space Spacing (blanks and tabs) in regular expressions are ignored. + --from=FILE+ Read additional pathnames of files to search from FILE. When FILE+ is a `-', standard input is read. This option is useful with+ `find+ -G, --basic-regexp Interpret patterns as basic regular expressions (BREs). -g GLOBS, --glob=GLOBS, --iglob=GLOBS Only search files whose name matches the specified comma-separated- list of GLOBS, same as --include=glob for each `glob' in GLOBS.+ list of GLOBS, same as --include='glob' for each `glob' in GLOBS. When a `glob' is preceded by a `!' or a `^', skip files whose name matches `glob', same as --exclude='glob'. When `glob' contains a- `/', full pathnames are matched. Otherwise basenames are matched.- When `glob' ends with a `/', directories are matched, same as- --include-dir='glob' and --exclude-dir='glob'. A leading `/'- matches the working directory. Option --iglob performs- case-insensitive name matching. This option may be repeated and- may be combined with options -M, -O and -t. For more details, see- `ugrep --help globs' and `man ugrep' section GLOBBING for details.+ `/', full pathnames are matched and option -r is enabled to+ recurse into directories if -d, -r or -R is not already specified.+ Otherwise basenames are matched. When `glob' ends with a `/',+ directories are matched, same as --include-dir='glob' and+ --exclude-dir='glob'. A leading `./' or just `/' matches the+ working directory. Option --iglob=GLOBS performs case-insensitive+ name matching. This option may be repeated and may be combined+ with options -M, -O and -t. For more details, see `ugrep --help+ globs' and `man ugrep' section GLOBBING for details. --glob-ignore-case Perform case-insensitive glob matching in general.@@ -4448,8 +4460,7 @@ in markdown: -i, --ignore-case Perform case insensitive matching. By default, ugrep is case- sensitive. By default, this option applies to ASCII letters only.- Use options -P and -i for Unicode case insensitive matching.+ sensitive. --ignore-files[=FILE] Ignore files and directories matching the globs in each FILE that@@ -4468,26 +4479,30 @@ in markdown: --include=GLOB Only search files whose name matches GLOB, same as -g GLOB. GLOB- can use **, *, ?, and [...] as wildcards and \ to quote a wildcard+ may use **, *, ?, and [...] as wildcards and \ to quote a wildcard or backslash character literally. When GLOB contains a `/', full- pathnames are matched. Otherwise basenames are matched. When- GLOB ends with a `/', directories are included as if --include-dir- is specified. Otherwise files are included. Note that --exclude- patterns take priority over --include patterns. GLOB should be- quoted to prevent shell globbing. This option may be repeated.+ pathnames are matched and option -r is enabled to recurse into+ directories if -d, -r or -R is not already specified. Otherwise+ basenames are matched. When GLOB ends with a `/', directories are+ included as if --include-dir is specified. Otherwise files are+ included. Note that --exclude patterns take priority over+ --include patterns. GLOB should be quoted to prevent shell+ globbing. This option may be repeated. --include-dir=GLOB Only directories whose name matches GLOB are included in recursive- searches, same as -g GLOB/. GLOB can use **, *, ?, and [...] as+ searches, same as -g GLOB/. GLOB may use **, *, ?, and [...] as wildcards and \ to quote a wildcard or backslash character- literally. When GLOB contains a `/', full pathnames are matched.- Otherwise basenames are matched. Note that --exclude-dir patterns- take priority over --include-dir patterns. GLOB should be quoted- to prevent shell globbing. This option may be repeated.+ literally. When GLOB contains a `/', full pathnames are matched+ and option -r is enabled to recurse into directories if -d, -r or+ -R is not already specified. Otherwise basenames are matched.+ Note that --exclude-dir patterns take priority over --include-dir+ patterns. GLOB should be quoted to prevent shell globbing. This+ option may be repeated. --include-from=FILE Read the globs from FILE and search only files and directories- whose name matches one or more globs. A glob can use **, *, ?,+ whose name matches one or more globs. A glob may use **, *, ?, and [...] as wildcards and \ to quote a wildcard or backslash character literally. When a glob contains a `/', full pathnames are matched. Otherwise basenames are matched. When a glob ends@@ -4516,8 +4531,8 @@ in markdown: that the start-up time to search may be increased when complex search patterns are specified that contain large Unicode character classes combined with `*' or `+' repeats, which should be avoided.- Option -U (--ascii) improves performance. Option --stats displays- an index search report.+ Option -U or --ascii improves performance. Option --stats+ displays an index search report. -J NUM, --jobs=NUM Specifies the number of threads spawned to search files. By@@ -4527,7 +4542,7 @@ in markdown: -j, --smart-case Perform case insensitive matching, unless a pattern is specified- with a literal upper case ASCII letter.+ with a literal upper case letter. --json Output file matches in JSON. When -H, -n, -k, or -b is specified, additional values are output. See also options --format and -u.@@ -4685,7 +4700,7 @@ in markdown: specified with -e PATTERN, i.e. a PATTERN argument requires option -e. Press F1 or CTRL-Z to view the help screen. Press F2 or CTRL-Y to invoke a command to view or edit the file shown at the- top of the screen. The command can be specified with option+ top of the screen. This command can be specified with option --view and defaults to environment variable PAGER when defined, or VISUAL or EDITOR. Press TAB or SHIFT-TAB to navigate directories and to select a file to search. Press ENTER to select lines to@@ -4733,9 +4748,9 @@ in markdown: the OPTIONS are saved that do not cause searches to fail when combined with other options. Additional options may be specified by editing the saved configuration file. A configuration file may- be modified manually to specify one or more config[=FILE] to- indirectly load the specified FILE, but recursive config loading- is not allowed.+ be modified by adding one or more config=FILE to include+ configuration files, but recursive configuration file inclusion is+ not permitted. --separator[=SEP], --context-separator=SEP Use SEP as field separator between file name, line number, column@@ -4791,9 +4806,9 @@ in markdown: `Pdf', `perl', `Perl', `php', `Php', `png', `Png', `prolog', `python', `Python', `r', `rpm', `Rpm', `rst', `rtf', `Rtf', `ruby', `Ruby', `rust', `scala', `scheme', `seed7', `shell',- `Shell', `smalltalk', `sql', `svg', `swift', `tcl', `tex',- `text', `tiff', `Tiff', `tt', `typescript', `verilog', `vhdl',- `vim', `xml', `Xml', `yacc', `yaml', `zig'.+ `Shell', `smalltalk', `sql', `svg', `swift', `tcl', `tex', `text',+ `tiff', `Tiff', `tt', `typescript', `verilog', `vhdl', `vim',+ `xml', `Xml', `yacc', `yaml', `zig'. --tabs[=NUM] Set the tab size to NUM to expand tabs for option -k. The value@@ -4864,10 +4879,11 @@ in markdown: -Y, --empty Empty-matching patterns match all lines. Normally, empty matches- are not output, unless a pattern begins with `^' or ends with `$'.- With this option, empty-matching patterns, such as x? and x*,- match all lines, not only lines with an `x' (enabled by default in- grep compatibility mode).+ are not output, unless a pattern starts with `^' or ends with `$'.+ With this option, empty-matching patterns, such as x*y?, x*y*z*,+ and x*(y?|z) match all lines, not only the expected pattern. This+ option is enabled by default in grep compatibility mode; specify+ option --no-empty to prevent matching all lines. -y, --any-line, --passthru Any line is output (passthru). Non-matching lines are output as@@ -4897,15 +4913,17 @@ in markdown: matching pathnames of files in archives are output in braces. When used with option --zmax=NUM, searches the contents of compressed files and archives stored within archives up to NUM- levels. When -g, -O, -M, or -t is specified, searches files- stored in archives whose filenames match globs, match filename- extensions, match file signature magic bytes, or match file types,- respectively. Supported compression formats: gzip (.gz), compress- (.Z), zip, 7z, bzip2 (requires suffix .bz, .bz2, .bzip2, .tbz,- .tbz2, .tb2, .tz2), lzma and xz (requires suffix .lzma, .tlz, .xz,- .txz), lz4 (requires suffix .lz4), zstd (requires suffix .zst,- .zstd, .tzst), brotli (requires suffix .br), bzip3 (requires- suffix .bz3).+ levels. When -g, -O, -M, or -t is specified, searches archives+ for files that match the specified globs, filename extensions,+ file signature magic bytes, or file types, respectively; a+ side-effect of these options is that the compressed files and+ archives searched are only those with filename extensions that+ match known compression and archive types. Supported compression+ formats: gzip (.gz), compress (.Z), zip, 7z, bzip2 (.bz, .bz2,+ .bzip2, .tbz, .tbz2, .tb2, .tz2), xz (.xz, .txz) and lzma+ (requires suffix .lzma, .tlz), zstd (.zst, .zstd, .tzst), lz4+ (requires suffix .lz4), brotli (requires suffix .br), bzip3+ (requires suffix .bz3). --zmax=NUM When used with option -z or --decompress, searches the contents of@@ -4931,10 +4949,10 @@ in markdown: status is 0 even if an error occurred. CONFIGURATION- The ug command is intended for context-dependent interactive searching- and is equivalent to the ugrep --config --pretty --sort command to load- the default configuration file `.ugrep' when present in the working- directory or in the home directory.+ The ug command is intended for interactive searching and is equivalent to+ the ugrep --config --pretty --sort command to load the `.ugrep`+ configuration file located in the working directory or, when not found,+ in the home directory. A configuration file contains `NAME=VALUE' pairs per line, where `NAME` is the name of a long option (without `--') and `=VALUE' is an argument,@@ -4943,8 +4961,8 @@ in markdown: The --config=FILE option and its abbreviated form ---FILE load the specified configuration file located in the working directory or, when- not found, located in the home directory. An error is produced when FILE- is not found or cannot be read.+ not found, in the home directory. An error is produced when FILE is not+ found or cannot be read. Command line options are parsed in the following order: the configuration file is loaded first, followed by the remaining options and arguments on@@ -4952,14 +4970,17 @@ in markdown: The --save-config option saves a `.ugrep' configuration file to the working directory with a subset of the options specified on the command- line. The --save-config=FILE option saves the configuration to FILE.- The configuration is written to standard output when FILE is a `-'.+ line. Only part of the specified options are saved that do not cause+ searches to fail when combined with other options. The --save-+ config=FILE option saves the configuration to FILE. The configuration is+ written to standard output when FILE is a `-'. GLOBBING- Globbing is used by options -g, --include, --include-dir, --include-from,- --exclude, --exclude-dir, --exclude-from and --ignore-files to match- pathnames and basenames in recursive searches. Glob arguments for these- options should be quoted to prevent shell globbing.+ Globbing is used by options -g (--glob and --iglob), --include,+ --include-dir, --include-from, --exclude, --exclude-dir, --exclude-from+ and --ignore-files to match pathnames or basenames in recursive searches.+ Glob arguments for these options should be quoted to prevent shell+ globbing. Globbing supports gitignore syntax and the corresponding matching rules, except that a glob normally matches files but not directories. If a glob@@ -4968,8 +4989,8 @@ in markdown: a path separator `/', the full pathname is matched. Otherwise the basename of a file or directory is matched. For example, *.h matches foo.h and bar/foo.h. bar/*.h matches bar/foo.h but not foo.h and not- bar/bar/foo.h. Use a leading `/' to force /*.h to match foo.h but not- bar/foo.h.+ bar/bar/foo.h. Use a leading `./' or just `/' to force /*.h to match+ foo.h in the working directory but not bar/foo.h. When a glob starts with a `^' or a `!' as in -g^GLOB, the match is negated. Likewise, a `!' (but not a `^') may be used with globs in the@@ -4992,7 +5013,7 @@ in markdown: [!abc-e] Matches one character not a,b,c,d,e,/. - / When used at the start of a glob, matches if pathname has no /.+ / When used at the start of a glob, matches the working directory. When used at the end of a glob, matches directories only. **/ Matches zero or more directories.@@ -5050,7 +5071,8 @@ in markdown: GREP_PATH May be used to specify a file path to pattern files. The file path is used by option -f to open a pattern file, when the pattern- file does not exist.+ file does not exist. Defaults to /usr/local/share/ugrep/patterns+ when GREP_PATH is undefined. GREP_COLOR May be used to specify ANSI SGR parameters to highlight matches@@ -5119,8 +5141,9 @@ in markdown: qb= TUI regex braces. FORMAT- Option --format=FORMAT specifies an output format for file matches.- Fields may be used in FORMAT, which expand into the following values:+ Option --format=FORMAT specifies an output format for matches. Option+ --replace=FORMAT only replaces the matching patterns. Fields may be used+ in FORMAT, which expand into the following values: %[TEXT]F if option -H is used: TEXT, the file pathname and separator.@@ -5339,11 +5362,11 @@ in markdown: List lines with `amount' and a decimal, ignoring case (space is AND): - $ ugrep -i -% 'amount +(.+)?' myfile.txt+ $ ugrep -i -% 'amount \d+(\.\d+)?' myfile.txt - Alternative query:+ The same Boolean search query, but with options -e and --and: - $ ugrep -wi -e amount --and '+(.+)?' myfile.txt+ $ ugrep -wi -e amount --and '\d+(\.\d+)?' myfile.txt List all Unicode words in a file: @@ -5369,7 +5392,6 @@ in markdown: $ ugrep -o -C20 -R -n -k -tjs FIXME - Find blocks of text between lines matching BEGIN and END by using a lazy quantifier `*?' to match only what is necessary and pattern `\n' to match newlines:@@ -5458,14 +5480,14 @@ in markdown: $ ugrep -3 -l -tRpm '' rpm/ - Monitor the system log for bug reports and ungroup multiple matches on a- line:+ Monitor the system log for bug reports and ungroup multiple matches from+ matching lines into separate lines: $ tail -f /var/log/system.log | ugrep -u -i -w bug Interactive fuzzy search with Boolean search queries: - $ ugrep -Q -l -% -Z3 --sort=best+ $ ugrep -Q -l -% -Z2 --sort=best Display all words in a MacRoman-encoded file that has CR newlines: @@ -5476,7 +5498,7 @@ in markdown: $ ugrep --help fuzzy COPYRIGHT- Copyright (c) 2021,2025 Robert A. van Engelen <engelen@acm.org>+ Copyright (c) 2021-2025 Robert A. van Engelen <engelen@acm.org> ugrep is released under the BSD-3 license. All parts of the software have reasonable copyright terms permitting free redistribution. This@@ -5490,7 +5512,7 @@ in markdown: - ugrep 7.2.2 February 3, 2025 UGREP(1)+ ugrep 7.4.0 April 21, 2025 UGREP(1) 🔝 [Back to table of contents](#toc) @@ -5765,3 +5787,4 @@ the [man page](#man). If you can't find it there and it looks like a bug, then reports are quickly addressed. *Copyright (c) Robert van Engelen, 2025*+
@@ -1,9 +1,9 @@ Windows 32 bit executables ========================== -`ugrep.exe` offers the same functionality as ugrep (and ug when renamed).+`ugrep.exe` offers the same functionality as ugrep described in the documentation. -`ug.exe` must be created manually by copying `ugrep.exe` to `ug.exe` (it is the same file, but renamed).+`ug.exe` is `ugrep.exe` renamed; behaves the same as the ug command (by means of naming it `ug.exe`). If you want ugrep to emulate GNU grep, then copy `ugrep.exe` to `grep.exe` and to `egrep.exe`, `fgrep.exe`.
@@ -1,10 +1,12 @@ Windows 64 bit executables ========================== -`ugrep.exe` offers the same functionality as ugrep (and ug when renamed).+`ugrep.exe` offers the same functionality as ugrep described in the documentation. -`ug.exe` must be created manually by copying `ugrep.exe` to `ug.exe` (it is the same file, but renamed).+`ug.exe` is `ugrep.exe` renamed; behaves the same as the ug command (by means of naming it `ug.exe`). If you want ugrep to emulate GNU grep, then copy `ugrep.exe` to `grep.exe` and to `egrep.exe`, `fgrep.exe`. +The executables expect and require CPUs supporting SSE2.+ The executables detect AVX2/AVX512BW automatically for optimized searching.
@@ -259,6 +259,7 @@ cat <<'END' --filter-magic-label=[+]LABEL:MAGIC Associate LABEL with files whose signature "magic bytes" match the MAGIC regex pattern. --format=FORMAT Output FORMAT-formatted matches. --free-space Spacing (blanks and tabs) in regular expressions are ignored.+--from=FILE Read additional pathnames of files to search from FILE. -G, --basic-regexp Interpret patterns as basic regular expressions (BREs). -g GLOBS, --glob=GLOBS, --iglob=GLOBS Only search files whose name matches the specified comma-separated list of GLOBS, same as --include=glob for each glob in GLOBS. --glob-ignore-case Perform case-insensitive glob matching in general.@@ -361,3 +362,4 @@ _comp_cmd_ugrep_encoding() COMPREPLY=( $(compgen -W "binary ASCII UTF-8 UTF-16 UTF-16BE UTF-16LE UTF-32 UTF-32BE UTF-32LE LATIN1 ISO-8859-1 ISO-8859-2 ISO-8859-3 ISO-8859-4 ISO-8859-5 ISO-8859-6 ISO-8859-7 ISO-8859-8 ISO-8859-9 ISO-8859-10 ISO-8859-11 ISO-8859-13 ISO-8859-14 ISO-8859-15 ISO-8859-16 MAC MACROMAN EBCDIC CP437 CP850 CP858 CP1250 CP1251 CP1252 CP1253 CP1254 CP1255 CP1256 CP1257 CP1258 KOI8-R KOI8-U KOI8-RU null-data" -- $cur) ) compopt +o nospace }+
@@ -37,6 +37,7 @@ complete -c ug+ -l filter -d 'Filter files through the specified COMMANDS first complete -c ug+ -l filter-magic-label -d 'Associate LABEL with files whose signature "magic bytes" match the MAGIC regex pattern' complete -c ug+ -l format -d 'Output FORMAT-formatted matches' complete -c ug+ -l free-space -d 'Spacing (blanks and tabs) in regular expressions are ignored'+complete -c ug+ -l from -d 'Read additional pathnames of files to search from FILE' complete -c ug+ -s G -l basic-regexp -d 'Interpret patterns as basic regular expressions (BREs)' complete -c ug+ -s g -r -l glob -l iglob -d 'Only search files whose name matches the specified comma-separated list of GLOBS, same as --include=glob for each glob in GLOBS' complete -c ug+ -l glob-ignore-case -d 'Perform case-insensitive glob matching in general'
@@ -37,6 +37,7 @@ complete -c ug -l filter -d 'Filter files through the specified COMMANDS first b complete -c ug -l filter-magic-label -d 'Associate LABEL with files whose signature "magic bytes" match the MAGIC regex pattern' complete -c ug -l format -d 'Output FORMAT-formatted matches' complete -c ug -l free-space -d 'Spacing (blanks and tabs) in regular expressions are ignored'+complete -c ug -l from -d 'Read additional pathnames of files to search from FILE' complete -c ug -s G -l basic-regexp -d 'Interpret patterns as basic regular expressions (BREs)' complete -c ug -s g -r -l glob -l iglob -d 'Only search files whose name matches the specified comma-separated list of GLOBS, same as --include=glob for each glob in GLOBS' complete -c ug -l glob-ignore-case -d 'Perform case-insensitive glob matching in general'
@@ -37,6 +37,7 @@ complete -c ugrep+ -l filter -d 'Filter files through the specified COMMANDS fir complete -c ugrep+ -l filter-magic-label -d 'Associate LABEL with files whose signature "magic bytes" match the MAGIC regex pattern' complete -c ugrep+ -l format -d 'Output FORMAT-formatted matches' complete -c ugrep+ -l free-space -d 'Spacing (blanks and tabs) in regular expressions are ignored'+complete -c ugrep+ -l from -d 'Read additional pathnames of files to search from FILE' complete -c ugrep+ -s G -l basic-regexp -d 'Interpret patterns as basic regular expressions (BREs)' complete -c ugrep+ -s g -r -l glob -l iglob -d 'Only search files whose name matches the specified comma-separated list of GLOBS, same as --include=glob for each glob in GLOBS' complete -c ugrep+ -l glob-ignore-case -d 'Perform case-insensitive glob matching in general'
@@ -37,6 +37,7 @@ complete -c ugrep -l filter -d 'Filter files through the specified COMMANDS firs complete -c ugrep -l filter-magic-label -d 'Associate LABEL with files whose signature "magic bytes" match the MAGIC regex pattern' complete -c ugrep -l format -d 'Output FORMAT-formatted matches' complete -c ugrep -l free-space -d 'Spacing (blanks and tabs) in regular expressions are ignored'+complete -c ugrep -l from -d 'Read additional pathnames of files to search from FILE' complete -c ugrep -s G -l basic-regexp -d 'Interpret patterns as basic regular expressions (BREs)' complete -c ugrep -s g -r -l glob -l iglob -d 'Only search files whose name matches the specified comma-separated list of GLOBS, same as --include=glob for each glob in GLOBS' complete -c ugrep -l glob-ignore-case -d 'Perform case-insensitive glob matching in general'
@@ -44,6 +44,7 @@ $matchers{-F,--fixed-strings}'[Interpret pattern as a set of fixed strings, sepa --filter-magic-label=-'[Associate LABEL with files whose signature "magic bytes" match the MAGIC regex pattern]' --format=-'[Output FORMAT-formatted matches]' --free-space'[Spacing (blanks and tabs) in regular expressions are ignored]'+--from=-'[Read additional pathnames of files to search from FILE]' $matchers{-G,--basic-regexp}'[Interpret patterns as basic regular expressions (BREs)]' '(1)*'{-g,--glob=-,--iglob=-}'[Only search files whose name matches the specified comma-separated list of GLOBS, same as --include=glob for each glob in GLOBS]:GLOB:( )' --glob-ignore-case'[Perform case-insensitive glob matching in general]'
@@ -44,6 +44,7 @@ $matchers{-F,--fixed-strings}'[Interpret pattern as a set of fixed strings, sepa --filter-magic-label=-'[Associate LABEL with files whose signature "magic bytes" match the MAGIC regex pattern]' --format=-'[Output FORMAT-formatted matches]' --free-space'[Spacing (blanks and tabs) in regular expressions are ignored]'+--from=-'[Read additional pathnames of files to search from FILE]' $matchers{-G,--basic-regexp}'[Interpret patterns as basic regular expressions (BREs)]' '(1)*'{-g,--glob=-,--iglob=-}'[Only search files whose name matches the specified comma-separated list of GLOBS, same as --include=glob for each glob in GLOBS]:GLOB:( )' --glob-ignore-case'[Perform case-insensitive glob matching in general]'
@@ -44,6 +44,7 @@ $matchers{-F,--fixed-strings}'[Interpret pattern as a set of fixed strings, sepa --filter-magic-label=-'[Associate LABEL with files whose signature "magic bytes" match the MAGIC regex pattern]' --format=-'[Output FORMAT-formatted matches]' --free-space'[Spacing (blanks and tabs) in regular expressions are ignored]'+--from=-'[Read additional pathnames of files to search from FILE]' $matchers{-G,--basic-regexp}'[Interpret patterns as basic regular expressions (BREs)]' '(1)*'{-g,--glob=-,--iglob=-}'[Only search files whose name matches the specified comma-separated list of GLOBS, same as --include=glob for each glob in GLOBS]:GLOB:( )' --glob-ignore-case'[Perform case-insensitive glob matching in general]'
@@ -44,6 +44,7 @@ $matchers{-F,--fixed-strings}'[Interpret pattern as a set of fixed strings, sepa --filter-magic-label=-'[Associate LABEL with files whose signature "magic bytes" match the MAGIC regex pattern]' --format=-'[Output FORMAT-formatted matches]' --free-space'[Spacing (blanks and tabs) in regular expressions are ignored]'+--from=-'[Read additional pathnames of files to search from FILE]' $matchers{-G,--basic-regexp}'[Interpret patterns as basic regular expressions (BREs)]' '(1)*'{-g,--glob=-,--iglob=-}'[Only search files whose name matches the specified comma-separated list of GLOBS, same as --include=glob for each glob in GLOBS]:GLOB:( )' --glob-ignore-case'[Perform case-insensitive glob matching in general]'
@@ -1,6 +1,6 @@ #! /bin/sh # Guess values for system-dependent variables and create Makefiles.-# Generated by GNU Autoconf 2.72 for ugrep 7.3.0.+# Generated by GNU Autoconf 2.72 for ugrep 7.4.0. # # Report bugs to <https://github.com/Genivia/ugrep/issues>. #@@ -606,8 +606,8 @@ MAKEFLAGS= # Identity of this package. PACKAGE_NAME='ugrep' PACKAGE_TARNAME='ugrep'-PACKAGE_VERSION='7.3.0'-PACKAGE_STRING='ugrep 7.3.0'+PACKAGE_VERSION='7.4.0'+PACKAGE_STRING='ugrep 7.4.0' PACKAGE_BUGREPORT='https://github.com/Genivia/ugrep/issues' PACKAGE_URL='https://ugrep.com' @@ -1382,7 +1382,7 @@ if test "$ac_init_help" = "long"; then # Omit some internal or obsolete options to make the list less imposing. # This message is too long to be a string in the A/UX 3.1 sh. cat <<_ACEOF-'configure' configures ugrep 7.3.0 to adapt to many kinds of systems.+'configure' configures ugrep 7.4.0 to adapt to many kinds of systems. Usage: $0 [OPTION]... [VAR=VALUE]... @@ -1453,7 +1453,7 @@ fi if test -n "$ac_init_help"; then case $ac_init_help in- short | recursive ) echo "Configuration of ugrep 7.3.0:";;+ short | recursive ) echo "Configuration of ugrep 7.4.0:";; esac cat <<\_ACEOF @@ -1630,7 +1630,7 @@ fi test -n "$ac_init_help" && exit $ac_status if $ac_init_version; then cat <<\_ACEOF-ugrep configure 7.3.0+ugrep configure 7.4.0 generated by GNU Autoconf 2.72 Copyright (C) 2023 Free Software Foundation, Inc.@@ -2184,7 +2184,7 @@ cat >config.log <<_ACEOF This file contains any messages produced by compilers while running configure, to aid debugging if configure makes a mistake. -It was created by ugrep $as_me 7.3.0, which was+It was created by ugrep $as_me 7.4.0, which was generated by GNU Autoconf 2.72. Invocation command line was $ $0$ac_configure_args_raw@@ -3694,7 +3694,7 @@ fi # Define the identity of the package. PACKAGE='ugrep'- VERSION='7.3.0'+ VERSION='7.4.0' printf "%s\n" "#define PACKAGE \"$PACKAGE\"" >>confdefs.h@@ -11808,7 +11808,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1 # report actual input values of CONFIG_FILES etc. instead of their # values after options handling. ac_log="-This file was extended by ugrep $as_me 7.3.0, which was+This file was extended by ugrep $as_me 7.4.0, which was generated by GNU Autoconf 2.72. Invocation command line was CONFIG_FILES = $CONFIG_FILES@@ -11881,7 +11881,7 @@ ac_cs_config_escaped=`printf "%s\n" "$ac_cs_config" | sed "s/^ //; s/'/'\\\\\\\\ cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 ac_cs_config='$ac_cs_config_escaped' ac_cs_version="\\-ugrep config.status 7.3.0+ugrep config.status 7.4.0 configured by $0, generated by GNU Autoconf 2.72, with options \\"\$ac_cs_config\\"
@@ -1,4 +1,4 @@-AC_INIT([ugrep],[7.3.0],[https://github.com/Genivia/ugrep/issues],[ugrep],[https://ugrep.com])+AC_INIT([ugrep],[7.4.0],[https://github.com/Genivia/ugrep/issues],[ugrep],[https://ugrep.com]) AM_INIT_AUTOMAKE([foreign subdir-objects dist-xz no-dist-gzip]) AC_CONFIG_HEADERS([config.h]) AC_COPYRIGHT([Copyright (C) 2019-2025 Robert van Engelen, Genivia Inc.])
@@ -138,7 +138,7 @@ class AbstractMatcher { num(num) { } const char *buf; ///< pointer to buffer- size_t len; ///< length of buffered context+ size_t len; ///< length of buffered context up to the matching line (bol_) size_t num; ///< number of bytes shifted out so far, when buffer shifted }; /// Event handler functor base class to invoke when the buffer contents are shifted out, e.g. for logging the data searched.@@ -471,6 +471,12 @@ class AbstractMatcher { { evh_ = handler; }+ /// Invoke handler explicitly (externally) with zero shift distance.+ inline void handle()+ {+ if (evh_)+ (*evh_)(*this, buf_, 0, num_);+ } /// Set reserved bytes for buffer shifting inline void set_reserve(size_t n) {@@ -503,7 +509,7 @@ class AbstractMatcher { DBGLOG("AbstractMatcher::interactive()"); (void)buffer(1); }- /// Flush the buffer's remaining content.+ /// Flush and delete the buffer's remaining content. inline void flush() { DBGLOG("AbstractMatcher::flush()");@@ -583,7 +589,6 @@ class AbstractMatcher { } return *this; }- /// Returns nonzero capture index (i.e. true) if the entire input matches this matcher's pattern (and internally caches the true/false result to permit repeat invocations). inline size_t matches() /// @returns nonzero capture index if the entire input matched this matcher's pattern, zero (i.e. false) otherwise@@ -674,7 +679,7 @@ class AbstractMatcher { return utf8(txt_); } #if WITH_SPAN- /// Set or reset mode to count matching lines only and skip other (e.g. for speed).+ /// Set or reset mode to count matching lines only and skip all other (e.g. for speed). inline void lineno_skip(bool f = false) { cml_ = f;@@ -1056,23 +1061,19 @@ class AbstractMatcher { #if WITH_FAST_GET return pos_ < end_ ? static_cast<unsigned char>(buf_[pos_]) : peek_more(); #else- if (pos_ < end_)- return static_cast<unsigned char>(buf_[pos_]);- if (eof_)- return EOF; while (true) {- if (end_ + blk_ + 1 >= max_)- (void)grow();- end_ += get(buf_ + end_, blk_ > 0 ? blk_ : max_ - end_ - 1); if (pos_ < end_) return static_cast<unsigned char>(buf_[pos_]);- DBGLOGN("peek(): EOF");- if (!wrap())- {- eof_ = true;+ if (eof_) return EOF;- }+ if (end_ + blk_ + 1 >= max_)+ (void)grow();+ size_t n = get(buf_ + end_, blk_ > 0 ? blk_ : max_ - end_ - 1);+ if (n == 0)+ eof_ = !wrap();+ else+ end_ += n; } #endif }@@ -1541,23 +1542,19 @@ class AbstractMatcher { #if WITH_FAST_GET return pos_ < end_ ? static_cast<unsigned char>(buf_[pos_++]) : get_more(); #else- if (pos_ < end_)- return static_cast<unsigned char>(buf_[pos_++]);- if (eof_)- return EOF; while (true) {- if (end_ + blk_ + 1 >= max_)- (void)grow();- end_ += get(buf_ + end_, blk_ > 0 ? blk_ : max_ - end_ - 1); if (pos_ < end_) return static_cast<unsigned char>(buf_[pos_++]);- DBGLOGN("get(): EOF");- if (!wrap())- {- eof_ = true;+ if (eof_) return EOF;- }+ if (end_ + blk_ + 1 >= max_)+ (void)grow();+ size_t n = get(buf_ + end_, blk_ > 0 ? blk_ : max_ - end_ - 1);+ if (n == 0)+ eof_ = !wrap();+ else+ end_ += n; } #endif }@@ -1594,49 +1591,43 @@ class AbstractMatcher { else txt_ = buf_; // old txt_ position was shifted out, set to the begin of the buffer }- /// Get the next character and grow the buffer to make more room if necessary.+ /// Get the next character and grow the buffer to make more room if necessary, unless EOF. inline int get_more() /// @returns the character read (unsigned char 0..255) or EOF (-1) { DBGLOG("AbstractMatcher::get_more()");- if (eof_)- return EOF;- while (true)+ while (!eof_) { if (end_ + blk_ + 1 >= max_) (void)grow();- end_ += get(buf_ + end_, blk_ > 0 ? blk_ : max_ - end_ - 1);+ size_t n = get(buf_ + end_, blk_ > 0 ? blk_ : max_ - end_ - 1);+ if (n == 0)+ eof_ = !wrap();+ else+ end_ += n; if (pos_ < end_) return static_cast<unsigned char>(buf_[pos_++]);- DBGLOGN("get_more(): EOF");- if (!wrap())- {- eof_ = true;- return EOF;- } }+ return EOF; }- /// Peek at the next character and grow the buffer to make more room if necessary.+ /// Peek at the next character and grow the buffer to make more room if necessary, unless EOF. inline int peek_more() /// @returns the character (unsigned char 0..255) or EOF (-1) { DBGLOG("AbstractMatcher::peek_more()");- if (eof_)- return EOF;- while (true)+ while (!eof_) { if (end_ + blk_ + 1 >= max_) (void)grow();- end_ += get(buf_ + end_, blk_ > 0 ? blk_ : max_ - end_ - 1);+ size_t n = get(buf_ + end_, blk_ > 0 ? blk_ : max_ - end_ - 1);+ if (n == 0)+ eof_ = !wrap();+ else+ end_ += n; if (pos_ < end_) return static_cast<unsigned char>(buf_[pos_]);- DBGLOGN("peek_more(): EOF");- if (!wrap())- {- eof_ = true;- return EOF;- } }+ return EOF; } Option opt_; ///< options for matcher engines char *buf_; ///< input character sequence buffer@@ -1662,7 +1653,7 @@ class AbstractMatcher { #endif size_t cno_; ///< column number count (cached) size_t num_; ///< character count of the input till bol_- size_t res_; ///< reserve bytes to keep in the buffer before bol_ when shifting, use only w/o evh_() set+ size_t res_; ///< reserve bytes to keep in the buffer before bol_ when shifting bool own_; ///< true if AbstractMatcher::buf_ was allocated and should be deleted bool eof_; ///< input has reached EOF bool mat_; ///< true if AbstractMatcher::matches() was successful
@@ -782,11 +782,11 @@ unrolled: // text insertions are allowed while (err_ < max_) {- ++err_;- ch = get(); // reached the end?+ ch = get(); if (ch == EOF) break;+ ++err_; // skip one (multibyte) char if (!bin_ && ch >= 0xC0) {@@ -1077,7 +1077,7 @@ unrolled: loc = e - buf_; set_current_and_peek_more(loc); loc = cur_;- if (loc >= end_)+ if (loc >= end_ && eof_) break; } }@@ -1098,7 +1098,7 @@ unrolled: loc = e - buf_; set_current_and_peek_more(loc); loc = cur_;- if (loc + pat_->len_ > end_)+ if (loc + pat_->len_ > end_ && eof_) break; } }
@@ -320,9 +320,13 @@ class Input { static const file_encoding_type null_data = 38; ///< swap NUL with LF static const file_encoding_type custom = 39; ///< custom code page };- /// FILE* handler functor base class to handle FILE* errors and non-blocking FILE* reads+ /// FILE* handler functor base class to analyze FILE* input, handle errors and non-blocking FILE* reads struct Handler {- virtual int operator()(FILE*) = 0;+ virtual size_t operator()(+ FILE*, ///< open file+ char*, ///< pointer to buffer with data read from file, may be changed by handler+ size_t ///< length of the buffered data, should be returned by the functor, or shorter to ignore or zero to force EOF and stop reading+ ) = 0; virtual ~Handler() { }; }; /// Stream buffer for reflex::Input, derived from std::streambuf.@@ -714,8 +718,17 @@ class Input { while (true) { size_t k = file_get(s, n);- if (k > 0 || feof(file_) || handler_ == NULL || (*handler_)(file_) == 0)+ // invoke handler when set, pass FILE* and buffer pointer and size, handler returns equal or lesser size+ if (handler_ != NULL)+ k = (*handler_)(file_, s, k);+ if (k > 0) return k;+ // if a non-recoverable error occurred, then return 0, permit non-blocking IO errors to continue+ if (errno != EAGAIN && errno != EWOULDBLOCK && errno != EINTR)+ return 0;+ // wait until non-blocking IO is ready for reading, or EOF+ if (!file_ready())+ return 0; } } if (istream_)@@ -758,11 +771,31 @@ class Input { void file_size(); /// Called by size() for a std::istream. void istream_size();- /// Implements get() on a FILE*.+ /// Implements get() on a FILE*, is non-blocking if file is non-blocking, get() blocks to read at least one byte. size_t file_get( char *s, ///< points to the string buffer to fill with input size_t n) ///< size of buffer pointed to by s ;+ /// Read n bytes into buffer s from file, block when IO is non-blocking, return false when fewer bytes read on failure or eof.+ bool file_read(+ char *s, ///< points to the string buffer to file with input+ size_t n) ///< size of the buffer pointed to by s+ {+ if (file_ == NULL)+ return false;+ while (true)+ {+ size_t k = ::fread(s, 1, n, file_);+ s += k;+ n -= k;+ if (n == 0)+ return true;+ if (!file_ready())+ return false;+ }+ }+ /// Wait for non-blocking file input to be ready and return true or return false for eof and errors, ignore EINTR+ bool file_ready(); /// Set FILE* handler void set_handler(Handler *handler) {
@@ -1427,9 +1427,9 @@ class Matcher : public PatternMatcher<reflex::Pattern> { bool simd_advance_string_pma_avx512bw(size_t loc); bool simd_advance_string_pmh_avx512bw(size_t loc); // String NEON metnods- bool simd_advance_string_neon(const char *&s, const char *e);- bool simd_advance_string_pma_neon(const char *&s, const char *e);- bool simd_advance_string_pmh_neon(const char *&s, const char *e);+ bool simd_advance_string_neon(const char *& s, const char *e);+ bool simd_advance_string_pma_neon(const char *& s, const char *e);+ bool simd_advance_string_pmh_neon(const char *& s, const char *e); // Fallback Boyer-Moore methods bool advance_string_bm(size_t loc); bool advance_string_bm_pma(size_t loc);
@@ -47,7 +47,7 @@ std::string regex_error::regex_error_message_code(regex_error_type code, const c "mismatched { }", "mismatched [ ]", "mismatched quotation",- "empty expression",+ "empty (sub)expression", "empty character class", "invalid character class", "invalid character class range",
@@ -47,6 +47,7 @@ # define fseeko _fseeki64 #else # include <unistd.h> // off_t, fstat()+# include <sys/select.h> #endif namespace reflex {@@ -659,91 +660,88 @@ void Input::file_init() #endif // assume plain (ASCII, binary or UTF-8 without BOM) content by default utfx_ = file_encoding::plain;- while (true)+ // check first UTF BOM byte using a blocking read+ if (file_read(utf8_, 1)) {- // check first UTF BOM byte- if (::fread(utf8_, 1, 1, file_) == 1)+ ulen_ = 1;+ if (utf8_[0] == '\0' || utf8_[0] == '\xef' || utf8_[0] == '\xfe' || utf8_[0] == '\xff') {- ulen_ = 1;- if (utf8_[0] == '\0' || utf8_[0] == '\xef' || utf8_[0] == '\xfe' || utf8_[0] == '\xff')+ // check second UTF BOM byte using a blocking read+ if (file_read(utf8_ + 1, 1)) {- // check second UTF BOM byte- if (::fread(utf8_ + 1, 1, 1, file_) == 1)+ ulen_ = 2;+ if (utf8_[0] == '\0' && utf8_[1] == '\0') // UTF-32 big endian BOM 0000XXXX? {- ulen_ = 2;- if (utf8_[0] == '\0' && utf8_[1] == '\0') // UTF-32 big endian BOM 0000XXXX?+ // check third UTF BOM byte using a blocking read+ if (file_read(utf8_ + 2, 2)) {- if (::fread(utf8_ + 2, 2, 1, file_) == 1)+ ulen_ = 4;+ if (utf8_[2] == '\xfe' && utf8_[3] == '\xff') // UTF-32 big endian BOM 0000FEFF? {- ulen_ = 4;- if (utf8_[2] == '\xfe' && utf8_[3] == '\xff') // UTF-32 big endian BOM 0000FEFF?- {- size_ = 0;- ulen_ = 0;- utfx_ = file_encoding::utf32be;- }+ size_ = 0;+ ulen_ = 0;+ utfx_ = file_encoding::utf32be; } }- else if (utf8_[0] == '\xfe' && utf8_[1] == '\xff') // UTF-16 big endian BOM FEFF?+ }+ else if (utf8_[0] == '\xfe' && utf8_[1] == '\xff') // UTF-16 big endian BOM FEFF?+ {+ size_ = 0;+ ulen_ = 0;+ utfx_ = file_encoding::utf16be;+ }+ else if (utf8_[0] == '\xff' && utf8_[1] == '\xfe') // UTF-16 or UTF-32 little endian BOM FFFEXXXX?+ {+ // check third UTF BOM byte using a blocking read+ if (file_read(utf8_ + 2, 2)) { size_ = 0;- ulen_ = 0;- utfx_ = file_encoding::utf16be;- }- else if (utf8_[0] == '\xff' && utf8_[1] == '\xfe') // UTF-16 or UTF-32 little endian BOM FFFEXXXX?- {- if (::fread(utf8_ + 2, 2, 1, file_) == 1)+ if (utf8_[2] == '\0' && utf8_[3] == '\0') // UTF-32 little endian BOM FFFE0000? {- size_ = 0;- if (utf8_[2] == '\0' && utf8_[3] == '\0') // UTF-32 little endian BOM FFFE0000?+ ulen_ = 0;+ utfx_ = file_encoding::utf32le;+ }+ else+ {+ int c = static_cast<unsigned char>(utf8_[2]) | static_cast<unsigned char>(utf8_[3]) << 8;+ if (c < 0x80) {- ulen_ = 0;- utfx_ = file_encoding::utf32le;+ uidx_ = 2;+ ulen_ = 1; } else {- int c = static_cast<unsigned char>(utf8_[2]) | static_cast<unsigned char>(utf8_[3]) << 8;- if (c < 0x80)- {- uidx_ = 2;- ulen_ = 1;- }- else+ if (c >= 0xD800 && c < 0xE000) {- if (c >= 0xD800 && c < 0xE000)- {- // UTF-16 surrogate pair- if (c < 0xDC00 && ::fread(utf8_, 2, 1, file_) == 1 && (static_cast<unsigned char>(utf8_[1]) & 0xFC) == 0xDC)- c = 0x010000 - 0xDC00 + ((c - 0xD800) << 10) + (static_cast<unsigned char>(utf8_[0]) | static_cast<unsigned char>(utf8_[1]) << 8);- else- c = REFLEX_NONCHAR;- }- ulen_ = static_cast<unsigned short>(utf8(c, utf8_)); // always a short unsigned int+ // UTF-16 surrogate pair+ if (c < 0xDC00 && file_read(utf8_, 2) && (static_cast<unsigned char>(utf8_[1]) & 0xFC) == 0xDC)+ c = 0x010000 - 0xDC00 + ((c - 0xD800) << 10) + (static_cast<unsigned char>(utf8_[0]) | static_cast<unsigned char>(utf8_[1]) << 8);+ else+ c = REFLEX_NONCHAR; }- utfx_ = file_encoding::utf16le;+ ulen_ = static_cast<unsigned short>(utf8(c, utf8_)); // always a short unsigned int }+ utfx_ = file_encoding::utf16le; } }- else if (utf8_[0] == '\xef' && utf8_[1] == '\xbb') // UTF-8 BOM EFBBXX?+ }+ else if (utf8_[0] == '\xef' && utf8_[1] == '\xbb') // UTF-8 BOM EFBBXX?+ {+ // check third UTF BOM byte using a blocking read+ if (file_read(utf8_ + 2, 1)) {- if (::fread(utf8_ + 2, 1, 1, file_) == 1)+ ulen_ = 3;+ if (utf8_[2] == '\xbf') // UTF-8 BOM EFBBBF? {- ulen_ = 3;- if (utf8_[2] == '\xbf') // UTF-8 BOM EFBBBF?- {- if (size_ >= 3)- size_ -= 3;- ulen_ = 0;- utfx_ = file_encoding::utf8;- }+ if (size_ >= 3)+ size_ -= 3;+ ulen_ = 0;+ utfx_ = file_encoding::utf8; } } } }- break; }- if (handler_ == NULL || feof(file_) || (*handler_)(file_) == 0)- break; } } @@ -770,7 +768,7 @@ size_t Input::file_get(char *s, size_t n) switch (utfx_) { case file_encoding::utf16be:- while (n > 0 && ::fread(buf, 2, 1, file_) == 1)+ while (n > 0 && ::fread(buf, 1, 1, file_) == 1 && file_read(reinterpret_cast<char*>(buf) + 1, 1)) { int c = buf[0] << 8 | buf[1]; if (c < 0x80)@@ -782,8 +780,8 @@ size_t Input::file_get(char *s, size_t n) { if (c >= 0xD800 && c < 0xE000) {- // UTF-16 surrogate pair- if (c < 0xDC00 && ::fread(buf + 2, 2, 1, file_) == 1 && (buf[2] & 0xFC) == 0xDC)+ // UTF-16 surrogate pair, complete the pair using a blocking read+ if (c < 0xDC00 && file_read(reinterpret_cast<char*>(buf) + 2, 2) && (buf[2] & 0xFC) == 0xDC) c = 0x010000 - 0xDC00 + ((c - 0xD800) << 10) + (buf[2] << 8 | buf[3]); else c = REFLEX_NONCHAR;@@ -809,7 +807,7 @@ size_t Input::file_get(char *s, size_t n) size_ -= t - s; return t - s; case file_encoding::utf16le:- while (n > 0 && ::fread(buf, 2, 1, file_) == 1)+ while (n > 0 && ::fread(buf, 1, 1, file_) == 1 && file_read(reinterpret_cast<char*>(buf) + 1, 1)) { int c = buf[0] | buf[1] << 8; if (c < 0x80)@@ -821,8 +819,8 @@ size_t Input::file_get(char *s, size_t n) { if (c >= 0xD800 && c < 0xE000) {- // UTF-16 surrogate pair- if (c < 0xDC00 && ::fread(buf + 2, 2, 1, file_) == 1 && (buf[3] & 0xFC) == 0xDC)+ // UTF-16 surrogate pair, complete the pair using a blocking read+ if (c < 0xDC00 && file_read(reinterpret_cast<char*>(buf) + 2, 2) && (buf[3] & 0xFC) == 0xDC) c = 0x010000 - 0xDC00 + ((c - 0xD800) << 10) + (buf[2] | buf[3] << 8); else c = REFLEX_NONCHAR;@@ -848,7 +846,7 @@ size_t Input::file_get(char *s, size_t n) size_ -= t - s; return t - s; case file_encoding::utf32be:- while (n > 0 && ::fread(buf, 4, 1, file_) == 1)+ while (n > 0 && ::fread(buf, 1, 1, file_) == 1 && file_read(reinterpret_cast<char*>(buf) + 1, 3)) { int c = buf[0] << 24 | buf[1] << 16 | buf[2] << 8 | buf[3]; if (c < 0x80)@@ -879,7 +877,7 @@ size_t Input::file_get(char *s, size_t n) size_ -= t - s; return t - s; case file_encoding::utf32le:- while (n > 0 && ::fread(buf, 4, 1, file_) == 1)+ while (n > 0 && ::fread(buf, 1, 1, file_) == 1 && file_read(reinterpret_cast<char*>(buf) + 1, 3)) { int c = buf[0] | buf[1] << 8 | buf[2] << 16 | buf[3] << 24; if (c < 0x80)@@ -1024,6 +1022,31 @@ size_t Input::file_get(char *s, size_t n) } } +bool Input::file_ready()+{+ if (file_ == NULL || feof(file_))+ return false;+#if (defined(__WIN32__) || defined(_WIN32) || defined(WIN32) || defined(_WIN64) || defined(__BORLANDC__)) && !defined(__CYGWIN__) && !defined(__MINGW32__) && !defined(__MINGW64__)+ return true;+#else+ while (true)+ {+ struct timeval tv;+ fd_set rfds, efds;+ FD_ZERO(&rfds);+ FD_ZERO(&efds);+ FD_SET(0, &efds);+ tv.tv_sec = 1;+ tv.tv_usec = 0;+ int r = ::select(fileno(file_) + 1, &rfds, NULL, &efds, &tv);+ if (r < 0 && errno != EINTR)+ return false;+ if (r > 0)+ return FD_ISSET(fileno(file_), &efds) == 0;+ }+#endif+}+ void Input::wstring_size() { unsigned int c;
@@ -30,7 +30,7 @@ @file matcher.cpp regex engine @brief RE/flex matcher engine @author Robert van Engelen - engelen@genivia.com-@copyright (c) 2016-2024, Robert van Engelen, Genivia Inc. All rights reserved.+@copyright (c) 2016-2025, Robert van Engelen, Genivia Inc. All rights reserved. @copyright (c) BSD-3 License - see LICENSE.txt */ @@ -983,7 +983,7 @@ bool Matcher::advance_pattern_pin1_one(size_t loc) loc = e - buf_; set_current_and_peek_more(loc); loc = cur_;- if (loc + 1 > end_)+ if (loc + 1 > end_ && eof_) return false; } }@@ -996,13 +996,15 @@ bool Matcher::advance_pattern_pin1_pma(size_t loc) const size_t min = pat_->min_; const uint16_t lcp = pat_->lcp_; const uint16_t lcs = pat_->lcs_;-#if defined(HAVE_AVX512BW) || defined(HAVE_AVX2) || defined(HAVE_SSE2)- __m128i vlcp = _mm_set1_epi8(chr[0]);- __m128i vlcs = _mm_set1_epi8(chr[1]);+ const char chr0 = chr[0];+ const char chr1 = chr[1]; while (true) { const char *s = buf_ + loc + lcp; const char *e = buf_ + end_ + lcp - min + 1;+#if defined(HAVE_AVX512BW) || defined(HAVE_AVX2) || defined(HAVE_SSE2)+ __m128i vlcp = _mm_set1_epi8(chr0);+ __m128i vlcs = _mm_set1_epi8(chr1); while (s <= e - 16) { __m128i vstrlcp = _mm_loadu_si128(reinterpret_cast<const __m128i*>(s));@@ -1013,31 +1015,19 @@ bool Matcher::advance_pattern_pin1_pma(size_t loc) while (mask != 0) { uint32_t offset = ctz(mask);- loc = s - lcp + offset - buf_;- if (loc + 4 > end_ || pat_->predict_match(&buf_[loc]))+ size_t k = s - lcp + offset - buf_;+ if (k + 4 > end_ || pat_->predict_match(&buf_[k])) {- set_current(loc);+ set_current(k); return true; } mask &= mask - 1; } s += 16; }- loc = s - lcp - buf_;- set_current_and_peek_more(loc);- loc = cur_;- if (loc + min > end_)- return false;- if (loc + min + 15 > end_)- break;- } #elif defined(HAVE_NEON)- uint8x16_t vlcp = vdupq_n_u8(chr[0]);- uint8x16_t vlcs = vdupq_n_u8(chr[1]);- while (true)- {- const char *s = buf_ + loc + lcp;- const char *e = buf_ + end_ + lcp - min + 1;+ uint8x16_t vlcp = vdupq_n_u8(chr0);+ uint8x16_t vlcs = vdupq_n_u8(chr1); while (s <= e - 16) { uint8x16_t vstrlcp = vld1q_u8(reinterpret_cast<const uint8_t*>(s));@@ -1048,66 +1038,53 @@ bool Matcher::advance_pattern_pin1_pma(size_t loc) uint64_t mask = vgetq_lane_u64(vmask64, 0); if (mask != 0) {- loc = s - lcp - buf_;+ size_t k = s - lcp - buf_; if (static_cast<uint32_t>(mask) == 0) { mask >>= 32;- loc += 4;+ k += 4; } do { if ((mask & 0xff)) {- if (loc + 4 > end_ || pat_->predict_match(&buf_[loc]))+ if (k + 4 > end_ || pat_->predict_match(&buf_[k])) {- set_current(loc);+ set_current(k); return true; } } mask >>= 8;- ++loc;+ ++k; } while (mask != 0); } mask = vgetq_lane_u64(vmask64, 1); if (mask != 0) {- loc = s - lcp + 8 - buf_;+ size_t k = s - lcp + 8 - buf_; if (static_cast<uint32_t>(mask) == 0) { mask >>= 32;- loc += 4;+ k += 4; } do { if ((mask & 0xff)) {- if (loc + 4 > end_ || pat_->predict_match(&buf_[loc]))+ if (k + 4 > end_ || pat_->predict_match(&buf_[k])) {- set_current(loc);+ set_current(k); return true; } } mask >>= 8;- ++loc;+ ++k; } while (mask != 0); } s += 16; }- loc = s - lcp - buf_;- set_current_and_peek_more(loc);- loc = cur_;- if (loc + min > end_)- return false;- if (loc + min + 15 > end_)- break;- } #endif- const char chr0 = chr[0];- const char chr1 = chr[1];- while (true)- {- const char *s = buf_ + loc + lcp;- const char *e = buf_ + end_;+ e = buf_ + end_; if (s < e && (s = static_cast<const char*>(std::memchr(s, chr0, e - s))) != NULL) { s -= lcp;@@ -1121,10 +1098,11 @@ bool Matcher::advance_pattern_pin1_pma(size_t loc) } else {- loc = std::max<size_t>(loc, e - lcp - buf_);+ if (e > buf_ + loc + lcp)+ loc = e - buf_ - lcp; set_current_and_peek_more(loc); loc = cur_;- if (loc + min > end_)+ if (loc + min > end_ && eof_) return false; } }@@ -1137,13 +1115,15 @@ bool Matcher::advance_pattern_pin1_pmh(size_t loc) const char *chr = pat_->chr_; const uint16_t lcp = pat_->lcp_; const uint16_t lcs = pat_->lcs_;-#if defined(HAVE_AVX512BW) || defined(HAVE_AVX2) || defined(HAVE_SSE2)- __m128i vlcp = _mm_set1_epi8(chr[0]);- __m128i vlcs = _mm_set1_epi8(chr[1]);+ const int chr0 = chr[0];+ const int chr1 = chr[1]; while (true) { const char *s = buf_ + loc + lcp; const char *e = buf_ + end_ + lcp - MIN + 1;+#if defined(HAVE_AVX512BW) || defined(HAVE_AVX2) || defined(HAVE_SSE2)+ __m128i vlcp = _mm_set1_epi8(chr0);+ __m128i vlcs = _mm_set1_epi8(chr1); while (s <= e - 16) { __m128i vstrlcp = _mm_loadu_si128(reinterpret_cast<const __m128i*>(s));@@ -1154,31 +1134,19 @@ bool Matcher::advance_pattern_pin1_pmh(size_t loc) while (mask != 0) { uint32_t offset = ctz(mask);- loc = s - lcp + offset - buf_;- if (pat_->predict_match(&buf_[loc], MIN))+ size_t k = s - lcp + offset - buf_;+ if (pat_->predict_match(&buf_[k], MIN)) {- set_current(loc);+ set_current(k); return true; } mask &= mask - 1; } s += 16; }- loc = s - lcp - buf_;- set_current_and_peek_more(loc);- loc = cur_;- if (loc + MIN > end_)- return false;- if (loc + MIN + 15 > end_)- break;- } #elif defined(HAVE_NEON)- uint8x16_t vlcp = vdupq_n_u8(chr[0]);- uint8x16_t vlcs = vdupq_n_u8(chr[1]);- while (true)- {- const char *s = buf_ + loc + lcp;- const char *e = buf_ + end_ + lcp - MIN + 1;+ uint8x16_t vlcp = vdupq_n_u8(chr0);+ uint8x16_t vlcs = vdupq_n_u8(chr1); while (s <= e - 16) { uint8x16_t vstrlcp = vld1q_u8(reinterpret_cast<const uint8_t*>(s));@@ -1189,66 +1157,53 @@ bool Matcher::advance_pattern_pin1_pmh(size_t loc) uint64_t mask = vgetq_lane_u64(vmask64, 0); if (mask != 0) {- loc = s - lcp - buf_;+ size_t k = s - lcp - buf_; if (static_cast<uint32_t>(mask) == 0) { mask >>= 32;- loc += 4;+ k += 4; } do { if ((mask & 0xff)) {- if (pat_->predict_match(&buf_[loc], MIN))+ if (pat_->predict_match(&buf_[k], MIN)) {- set_current(loc);+ set_current(k); return true; } } mask >>= 8;- ++loc;+ ++k; } while (mask != 0); } mask = vgetq_lane_u64(vmask64, 1); if (mask != 0) {- loc = s - lcp + 8 - buf_;+ size_t k = s - lcp + 8 - buf_; if (static_cast<uint32_t>(mask) == 0) { mask >>= 32;- loc += 4;+ k += 4; } do { if ((mask & 0xff)) {- if (pat_->predict_match(&buf_[loc], MIN))+ if (pat_->predict_match(&buf_[k], MIN)) {- set_current(loc);+ set_current(k); return true; } } mask >>= 8;- ++loc;+ ++k; } while (mask != 0); } s += 16; }- loc = s - lcp - buf_;- set_current_and_peek_more(loc);- loc = cur_;- if (loc + MIN > end_)- return false;- if (loc + MIN + 15 > end_)- break;- } #endif- const int chr0 = chr[0];- const int chr1 = chr[1];- while (true)- {- const char *s = buf_ + loc + lcp;- const char *e = buf_ + end_;+ e = buf_ + end_; if (s < e && (s = static_cast<const char*>(std::memchr(s, chr0, e - s))) != NULL) { s -= lcp;@@ -1262,10 +1217,11 @@ bool Matcher::advance_pattern_pin1_pmh(size_t loc) } else {- loc = std::max<size_t>(loc, e - lcp - buf_);+ if (e > buf_ + loc + lcp)+ loc = e - buf_ - lcp; set_current_and_peek_more(loc); loc = cur_;- if (loc + MIN > end_)+ if (loc + MIN > end_ && eof_) return false; } }@@ -1292,25 +1248,31 @@ bool Matcher::advance_pattern_pin##N##_one(size_t loc) \ while (mask != 0) \ { \ uint32_t offset = ctz(mask); \- loc = s + offset - buf_; \- if (loc + 4 > end_ || pat_->predict_match(&buf_[loc])) \+ size_t k = s + offset - buf_; \+ if (k + 4 > end_ || pat_->predict_match(&buf_[k])) \ { \- set_current(loc); \+ set_current(k); \ return true; \ } \ mask &= mask - 1; \ } \ s += 16; \ } \+ while (s < e - 3) \+ { \+ if (pat_->predict_match(s++)) \+ { \+ size_t k = s - buf_ - 1; \+ set_current(k); \+ return true; \+ } \+ } \ loc = s - buf_; \ set_current_and_peek_more(loc); \ loc = cur_; \- if (loc + 1 > end_) \- return false; \- if (loc + 16 > end_) \- break; \+ if (loc + 3 >= end_) \+ return true; \ } \- return advance_pattern_min1(loc); \ } ADV_PAT_PIN_ONE(2, \@@ -1428,25 +1390,35 @@ bool Matcher::advance_pattern_pin##N##_pma(size_t loc) \ while (mask != 0) \ { \ uint32_t offset = ctz(mask); \- loc = s - lcp + offset - buf_; \- if (loc + min + 3 > end_ || pat_->predict_match(&buf_[loc])) \+ size_t k = s - lcp + offset - buf_; \+ if (k + min + 3 > end_ || pat_->predict_match(&buf_[k])) \ { \- set_current(loc); \+ set_current(k); \ return true; \ } \ mask &= mask - 1; \ } \ s += 16; \ } \- loc = s - lcp - buf_; \+ s -= lcp; \+ e = buf_ + end_ - 3; \+ while (s < e) \+ { \+ if (pat_->predict_match(s++)) \+ { \+ size_t k = s - buf_ - 1; \+ set_current(k); \+ return true; \+ } \+ } \+ loc = s - buf_; \ set_current_and_peek_more(loc); \ loc = cur_; \- if (loc + min > end_) \+ if (loc + min > end_ && eof_) \ return false; \- if (loc + min + 15 > end_) \- break; \+ if (loc + 3 >= end_) \+ return true; \ } \- return advance_pattern_pma(loc); \ } \ \ template <uint8_t MIN> \@@ -1471,25 +1443,33 @@ bool Matcher::advance_pattern_pin##N##_pmh(size_t loc) \ while (mask != 0) \ { \ uint32_t offset = ctz(mask); \- loc = s - lcp + offset - buf_; \- if (pat_->predict_match(&buf_[loc], MIN)) \+ size_t k = s - lcp + offset - buf_; \+ if (pat_->predict_match(&buf_[k], MIN)) \ { \- set_current(loc); \+ set_current(k); \ return true; \ } \ mask &= mask - 1; \ } \ s += 16; \ } \- loc = s - lcp - buf_; \+ s -= lcp; \+ e = buf_ + end_ - MIN + 1; \+ while (s < e) \+ { \+ if (pat_->predict_match(s++, MIN)) \+ { \+ size_t k = s - buf_ - 1; \+ set_current(k); \+ return true; \+ } \+ } \+ loc = s - buf_; \ set_current_and_peek_more(loc); \ loc = cur_; \- if (loc + MIN > end_) \+ if (loc + MIN > end_ && eof_) \ return false; \- if (loc + MIN + 15 > end_) \- break; \ } \- return advance_pattern_min4<MIN>(loc); \ } ADV_PAT_PIN(2, \@@ -1666,60 +1646,66 @@ bool Matcher::advance_pattern_pin##N##_one(size_t loc) \ uint64_t mask = vgetq_lane_u64(vmask64, 0); \ if (mask != 0) \ { \- loc = s - buf_; \+ size_t k = s - buf_; \ if (static_cast<uint32_t>(mask) == 0) \ { \ mask >>= 32; \- loc += 4; \+ k += 4; \ } \ do \ { \ if ((mask & 0xff) != 0) \ { \- if (loc + 4 > end_ || pat_->predict_match(&buf_[loc])) \+ if (k + 4 > end_ || pat_->predict_match(&buf_[k])) \ { \- set_current(loc); \+ set_current(k); \ return true; \ } \ } \ mask >>= 8; \- ++loc; \+ ++k; \ } while (mask != 0); \ } \ mask = vgetq_lane_u64(vmask64, 1); \ if (mask != 0) \ { \- loc = s + 8 - buf_; \+ size_t k = s + 8 - buf_; \ if (static_cast<uint32_t>(mask) == 0) \ { \ mask >>= 32; \- loc += 4; \+ k += 4; \ } \ do \ { \ if ((mask & 0xff) != 0) \ { \- if (loc + 4 > end_ || pat_->predict_match(&buf_[loc])) \+ if (k + 4 > end_ || pat_->predict_match(&buf_[k])) \ { \- set_current(loc); \+ set_current(k); \ return true; \ } \ } \ mask >>= 8; \- ++loc; \+ ++k; \ } while (mask != 0); \ } \ s += 16; \ } \+ while (s < e - 3) \+ { \+ if (pat_->predict_match(s++)) \+ { \+ size_t k = s - buf_ - 1; \+ set_current(k); \+ return true; \+ } \+ } \ loc = s - buf_; \ set_current_and_peek_more(loc); \ loc = cur_; \- if (loc + 1 > end_) \- return false; \- if (loc + 16 > end_) \- break; \+ if (loc + 3 >= end_) \+ return true; \ } \- return advance_pattern_min1(loc); \ } ADV_PAT_PIN_ONE(2, \@@ -1874,60 +1860,70 @@ bool Matcher::advance_pattern_pin##N##_pma(size_t loc) \ uint64_t mask = vgetq_lane_u64(vmask64, 0); \ if (mask != 0) \ { \- loc = s - lcp - buf_; \+ size_t k = s - lcp - buf_; \ if (static_cast<uint32_t>(mask) == 0) \ { \ mask >>= 32; \- loc += 4; \+ k += 4; \ } \ do \ { \ if ((mask & 0xff) != 0) \ { \- if (loc + 4 > end_ || pat_->predict_match(&buf_[loc])) \+ if (k + 4 > end_ || pat_->predict_match(&buf_[k])) \ { \- set_current(loc); \+ set_current(k); \ return true; \ } \ } \ mask >>= 8; \- ++loc; \+ ++k; \ } while (mask != 0); \ } \ mask = vgetq_lane_u64(vmask64, 1); \ if (mask != 0) \ { \- loc = s - lcp + 8 - buf_; \+ size_t k = s - lcp + 8 - buf_; \ if (static_cast<uint32_t>(mask) == 0) \ { \ mask >>= 32; \- loc += 4; \+ k += 4; \ } \ do \ { \ if ((mask & 0xff) != 0) \ { \- if (loc + 4 > end_ || pat_->predict_match(&buf_[loc])) \+ if (k + 4 > end_ || pat_->predict_match(&buf_[k])) \ { \- set_current(loc); \+ set_current(k); \ return true; \ } \ } \ mask >>= 8; \- ++loc; \+ ++k; \ } while (mask != 0); \ } \ s += 16; \ } \- loc = s - lcp - buf_; \+ s -= lcp; \+ e = buf_ + end_ - 3; \+ while (s < e) \+ { \+ if (pat_->predict_match(s++)) \+ { \+ size_t k = s - buf_ - 1; \+ set_current(k); \+ return true; \+ } \+ } \+ loc = s - buf_; \ set_current_and_peek_more(loc); \ loc = cur_; \- if (loc + min > end_) \+ if (loc + min > end_ && eof_) \ return false; \- if (loc + min + 15 > end_) \- break; \+ if (loc + 3 >= end_) \+ return true; \ } \- return advance_pattern_pma(loc); \ } \ \ template <uint8_t MIN> \@@ -1950,60 +1946,68 @@ bool Matcher::advance_pattern_pin##N##_pmh(size_t loc) \ uint64_t mask = vgetq_lane_u64(vmask64, 0); \ if (mask != 0) \ { \- loc = s - lcp - buf_; \+ size_t k = s - lcp - buf_; \ if (static_cast<uint32_t>(mask) == 0) \ { \ mask >>= 32; \- loc += 4; \+ k += 4; \ } \ do \ { \ if ((mask & 0xff) != 0) \ { \- if (loc + 4 > end_ || pat_->predict_match(&buf_[loc])) \+ if (k + 4 > end_ || pat_->predict_match(&buf_[k])) \ { \- set_current(loc); \+ set_current(k); \ return true; \ } \ } \ mask >>= 8; \- ++loc; \+ ++k; \ } while (mask != 0); \ } \ mask = vgetq_lane_u64(vmask64, 1); \ if (mask != 0) \ { \- loc = s - lcp + 8 - buf_; \+ size_t k = s - lcp + 8 - buf_; \ if (static_cast<uint32_t>(mask) == 0) \ { \ mask >>= 32; \- loc += 4; \+ k += 4; \ } \ do \ { \ if ((mask & 0xff) != 0) \ { \- if (loc + 4 > end_ || pat_->predict_match(&buf_[loc])) \+ if (k + 4 > end_ || pat_->predict_match(&buf_[k])) \ { \- set_current(loc); \+ set_current(k); \ return true; \ } \ } \ mask >>= 8; \- ++loc; \+ ++k; \ } while (mask != 0); \ } \ s += 16; \ } \- loc = s - lcp - buf_; \+ s -= lcp; \+ e = buf_ + end_ - MIN + 1; \+ while (s < e) \+ { \+ if (pat_->predict_match(s++, MIN)) \+ { \+ size_t k = s - buf_ - 1; \+ set_current(k); \+ return true; \+ } \+ } \+ loc = s - buf_; \ set_current_and_peek_more(loc); \ loc = cur_; \- if (loc + MIN > end_) \+ if (loc + MIN > end_ && eof_) \ return false; \- if (loc + MIN + 15 > end_) \- break; \ } \- return advance_pattern_min4<MIN>(loc); \ } ADV_PAT_PIN(2, \@@ -2273,8 +2277,8 @@ bool Matcher::advance_pattern_min1(size_t loc) loc = s - buf_ + 1; continue; }- loc = s - buf_;- set_current(loc);+ size_t k = s - buf_;+ set_current(k); return true; } loc = s - buf_;@@ -2316,15 +2320,15 @@ bool Matcher::advance_pattern_min2(size_t loc) c0 = c1; if ((state & 2) == 0 && (s > e - 2 || pat_->predict_match(s - 2))) {- loc = s - buf_ - 2;- set_current(loc);+ size_t k = s - buf_ - 2;+ set_current(k); return true; } } loc = s - buf_; set_current_and_peek_more(loc); loc = cur_;- if (loc + 1 >= end_)+ if (loc + 1 >= end_ && eof_) { uint8_t c1 = 0; // reached the end state = (state << 1) | tap[Pattern::bihash(c0, c1)];@@ -2355,15 +2359,15 @@ bool Matcher::advance_pattern_min3(size_t loc) c0 = c1; if ((state & 4) == 0 && (s > e - 1 || pat_->predict_match(s - 3))) {- loc = s - buf_ - 3;- set_current(loc);+ size_t k = s - buf_ - 3;+ set_current(k); return true; } } loc = s - buf_; set_current_and_peek_more(loc); loc = cur_;- if (loc + 1 >= end_)+ if (loc + 1 >= end_ && eof_) { uint8_t c1 = 0; // reached the end state = (state << 1) | tap[Pattern::bihash(c0, c1)];@@ -2398,14 +2402,14 @@ bool Matcher::advance_pattern_min4(size_t loc) state1 = (state2 << 1) | tap[Pattern::bihash(c1, c0)]; if ((state2 & mask) == 0 && pat_->predict_match(s - MIN - 1, MIN)) {- loc = s - buf_ - MIN - 1;- set_current(loc);+ size_t k = s - buf_ - MIN - 1;+ set_current(k); return true; } if ((state1 & mask) == 0 && pat_->predict_match(s - MIN, MIN)) {- loc = s - buf_ - MIN;- set_current(loc);+ size_t k = s - buf_ - MIN;+ set_current(k); return true; } }@@ -2413,7 +2417,7 @@ bool Matcher::advance_pattern_min4(size_t loc) set_current_and_peek_more(ahead); s = buf_ + cur_; e = buf_ + end_ - 2;- if (s >= e)+ if (s >= e && eof_) { uint8_t c0 = static_cast<uint8_t>(*s); if (s == e + 1)@@ -2422,8 +2426,8 @@ bool Matcher::advance_pattern_min4(size_t loc) state2 = (state1 << 1) | tap[Pattern::bihash(c0, c1)]; if ((state2 & mask) == 0 && pat_->predict_match(s - MIN + 1, MIN)) {- loc = s - buf_ - MIN + 1;- set_current(loc);+ size_t k = s - buf_ - MIN + 1;+ set_current(k); return true; } }@@ -2435,14 +2439,14 @@ bool Matcher::advance_pattern_min4(size_t loc) state1 = (state2 << 1) | tap[Pattern::bihash(c1, c0)]; if ((state2 & mask) == 0 && pat_->predict_match(s - MIN, MIN)) {- loc = s - buf_ - MIN;- set_current(loc);+ size_t k = s - buf_ - MIN;+ set_current(k); return true; } if ((state1 & mask) == 0 && pat_->predict_match(s - MIN + 1, MIN)) {- loc = s - buf_ - MIN + 1;- set_current(loc);+ size_t k = s - buf_ - MIN + 1;+ set_current(k); return true; } }@@ -2465,8 +2469,8 @@ bool Matcher::advance_pattern_pma(size_t loc) pat_->predict_match(s++) || pat_->predict_match(s++)) {- loc = s - buf_ - 1;- set_current(loc);+ size_t k = s - buf_ - 1;+ set_current(k); return true; } }@@ -2489,14 +2493,14 @@ bool Matcher::advance_char(size_t loc) s = static_cast<const char*>(std::memchr(s, chr0, e - s)); if (s != NULL) {- loc = s - buf_;- set_current(loc);+ size_t k = s - buf_;+ set_current(k); return true; } loc = e - buf_; set_current_and_peek_more(loc); loc = cur_;- if (loc + 1 > end_)+ if (loc + 1 > end_ && eof_) return false; } }@@ -2525,7 +2529,7 @@ bool Matcher::advance_char_pma(size_t loc) loc = e - buf_; set_current_and_peek_more(loc); loc = cur_;- if (loc + 1 > end_)+ if (loc + 1 > end_ && eof_) return false; } }@@ -2556,24 +2560,24 @@ bool Matcher::advance_char_pmh(size_t loc) loc = e - buf_; set_current_and_peek_more(loc); loc = cur_;- if (loc + 1 > end_)+ if (loc + 1 > end_ && eof_) return false; } } } -/// Few chars+/// Few (2 or 3) chars template <uint8_t LEN> bool Matcher::advance_chars(size_t loc) { const uint16_t lcp = 0; const uint16_t lcs = LEN - 1; const char *chr = pat_->chr_;-#if defined(HAVE_AVX512BW) || defined(HAVE_AVX2) || defined(HAVE_SSE2) while (true) { const char *s = buf_ + loc + lcp; const char *e = buf_ + end_ + lcp - LEN + 1;+#if defined(HAVE_AVX512BW) || defined(HAVE_AVX2) || defined(HAVE_SSE2) __m128i vlcp = _mm_set1_epi8(chr[lcp]); __m128i vlcs = _mm_set1_epi8(chr[lcs]); while (s <= e - 16)@@ -2586,30 +2590,18 @@ bool Matcher::advance_chars(size_t loc) while (mask != 0) { uint32_t offset = ctz(mask);- loc = s - lcp + offset - buf_;+ size_t k = s - lcp + offset - buf_; if (LEN == 2 || (LEN == 3 ? s[offset + 1 - lcp] == chr[1] : std::memcmp(s + 1 - lcp + offset, chr + 1, LEN - 1) == 0)) {- set_current(loc);+ set_current(k); return true; } mask &= mask - 1; } s += 16; }- loc = s - lcp - buf_;- set_current_and_peek_more(loc);- loc = cur_;- if (loc + LEN > end_)- return false;- if (loc + LEN + 15 > end_)- break;- } #elif defined(HAVE_NEON)- while (true)- {- const char *s = buf_ + loc + lcp;- const char *e = buf_ + end_ + lcp - LEN + 1; uint8x16_t vlcp = vdupq_n_u8(chr[lcp]); uint8x16_t vlcs = vdupq_n_u8(chr[lcs]); while (s <= e - 16)@@ -2623,62 +2615,50 @@ bool Matcher::advance_chars(size_t loc) uint64_t mask = vgetq_lane_u64(vmask64, 0); if (mask != 0) {- loc = s - lcp - buf_;+ size_t k = s - lcp - buf_; if (static_cast<uint32_t>(mask) == 0) { mask >>= 32;- loc += 4;+ k += 4; } do { if ((mask & 0xff) && (LEN == 2 ||- (LEN == 3 ? buf_[loc + 1] == chr[1] : std::memcmp(&buf_[loc + 1], chr + 1, LEN - 1) == 0)))+ (LEN == 3 ? buf_[k + 1] == chr[1] : std::memcmp(&buf_[k + 1], chr + 1, LEN - 1) == 0))) {- set_current(loc);+ set_current(k); return true; } mask >>= 8;- ++loc;+ ++k; } while (mask != 0); } mask = vgetq_lane_u64(vmask64, 1); if (mask != 0) {- loc = s - lcp + 8 - buf_;+ size_t k = s - lcp + 8 - buf_; if (static_cast<uint32_t>(mask) == 0) { mask >>= 32;- loc += 4;+ k += 4; } do { if ((mask & 0xff) && (LEN == 2 ||- (LEN == 3 ? buf_[loc + 1] == chr[1] : std::memcmp(&buf_[loc + 1], chr + 1, LEN - 1) == 0)))+ (LEN == 3 ? buf_[k + 1] == chr[1] : std::memcmp(&buf_[k + 1], chr + 1, LEN - 1) == 0))) {- set_current(loc);+ set_current(k); return true; } mask >>= 8;- ++loc;+ ++k; } while (mask != 0); } s += 16; }- loc = s - lcp - buf_;- set_current_and_peek_more(loc);- loc = cur_;- if (loc + LEN > end_)- return false;- if (loc + LEN + 15 > end_)- break;- } #endif- while (true)- {- const char *s = buf_ + loc + lcp;- const char *e = buf_ + end_ + lcp - LEN + 1; while (s < e) { do@@ -2692,8 +2672,8 @@ bool Matcher::advance_chars(size_t loc) if (LEN == 2 || (LEN == 3 ? s[1 - lcp] == chr[1] : std::memcmp(s + 1 - lcp, chr + 1, LEN - 1) == 0)) {- loc = s - lcp - buf_;- set_current(loc);+ size_t k = s - lcp - buf_;+ set_current(k); return true; } ++s;@@ -2701,12 +2681,12 @@ bool Matcher::advance_chars(size_t loc) loc = s - lcp - buf_; set_current_and_peek_more(loc); loc = cur_;- if (loc + LEN > end_)+ if (loc + LEN > end_ && eof_) return false; } } -/// Few chars followed by 1 to 4 characters+/// Few (2 or 3) chars followed by 1 to 4 characters template<uint8_t LEN> bool Matcher::advance_chars_pma(size_t loc) {@@ -2714,11 +2694,11 @@ bool Matcher::advance_chars_pma(size_t loc) const uint16_t lcs = LEN - 1; const char *chr = pat_->chr_; const size_t min = pat_->min_;-#if defined(HAVE_AVX512BW) || defined(HAVE_AVX2) || defined(HAVE_SSE2) while (true) { const char *s = buf_ + loc + lcp; const char *e = buf_ + end_ + lcp - LEN - min + 1;+#if defined(HAVE_AVX512BW) || defined(HAVE_AVX2) || defined(HAVE_SSE2) __m128i vlcp = _mm_set1_epi8(chr[lcp]); __m128i vlcs = _mm_set1_epi8(chr[lcs]); while (s <= e - 16)@@ -2734,10 +2714,10 @@ bool Matcher::advance_chars_pma(size_t loc) if (LEN == 2 || (LEN == 3 ? s[offset + 1 - lcp] == chr[1] : std::memcmp(s + 1 - lcp + offset, chr + 1, LEN - 1) == 0)) {- loc = s - lcp + offset - buf_;- if (loc + LEN + 4 > end_ || pat_->predict_match(&buf_[loc + LEN]))+ size_t k = s - lcp + offset - buf_;+ if (k + LEN + 4 > end_ || pat_->predict_match(&buf_[k + LEN])) {- set_current(loc);+ set_current(k); return true; } }@@ -2745,19 +2725,7 @@ bool Matcher::advance_chars_pma(size_t loc) } s += 16; }- loc = s - lcp - buf_;- set_current_and_peek_more(loc);- loc = cur_;- if (loc + LEN + min > end_)- return false;- if (loc + LEN + min + 15 > end_)- break;- } #elif defined(HAVE_NEON)- while (true)- {- const char *s = buf_ + loc + lcp;- const char *e = buf_ + end_ + lcp - LEN - min + 1; uint8x16_t vlcp = vdupq_n_u8(chr[lcp]); uint8x16_t vlcs = vdupq_n_u8(chr[lcs]); while (s <= e - 16)@@ -2771,68 +2739,56 @@ bool Matcher::advance_chars_pma(size_t loc) uint64_t mask = vgetq_lane_u64(vmask64, 0); if (mask != 0) {- loc = s - lcp - buf_;+ size_t k = s - lcp - buf_; if (static_cast<uint32_t>(mask) == 0) { mask >>= 32;- loc += 4;+ k += 4; } do { if ((mask & 0xff) && (LEN == 2 ||- (LEN == 3 ? buf_[loc + 1] == chr[1] : std::memcmp(&buf_[loc + 1], chr + 1, LEN - 1) == 0)))+ (LEN == 3 ? buf_[k + 1] == chr[1] : std::memcmp(&buf_[k + 1], chr + 1, LEN - 1) == 0))) {- if (loc + LEN + 4 > end_ || pat_->predict_match(&buf_[loc + LEN]))+ if (k + LEN + 4 > end_ || pat_->predict_match(&buf_[k + LEN])) {- set_current(loc);+ set_current(k); return true; } } mask >>= 8;- ++loc;+ ++k; } while (mask != 0); } mask = vgetq_lane_u64(vmask64, 1); if (mask != 0) {- loc = s - lcp + 8 - buf_;+ size_t k = s - lcp + 8 - buf_; if (static_cast<uint32_t>(mask) == 0) { mask >>= 32;- loc += 4;+ k += 4; } do { if ((mask & 0xff) && (LEN == 2 ||- (LEN == 3 ? buf_[loc + 1] == chr[1] : std::memcmp(&buf_[loc + 1], chr + 1, LEN - 1) == 0)))+ (LEN == 3 ? buf_[k + 1] == chr[1] : std::memcmp(&buf_[k + 1], chr + 1, LEN - 1) == 0))) {- if (loc + LEN + 4 > end_ || pat_->predict_match(&buf_[loc + LEN]))+ if (k + LEN + 4 > end_ || pat_->predict_match(&buf_[k + LEN])) {- set_current(loc);+ set_current(k); return true; } } mask >>= 8;- ++loc;+ ++k; } while (mask != 0); } s += 16; }- loc = s - lcp - buf_;- set_current_and_peek_more(loc);- loc = cur_;- if (loc + LEN + min > end_)- return false;- if (loc + LEN + min + 15 > end_)- break;- } #endif- while (true)- {- const char *s = buf_ + loc + lcp;- const char *e = buf_ + end_ + lcp - LEN - min + 1; while (s < e) { do@@ -2846,10 +2802,10 @@ bool Matcher::advance_chars_pma(size_t loc) if (LEN == 2 || (LEN == 3 ? s[1 - lcp] == chr[1] : std::memcmp(s + 1 - lcp, chr + 1, LEN - 1) == 0)) {- loc = s - lcp - buf_;- if (loc + LEN + 4 > end_ || pat_->predict_match(&buf_[loc + LEN]))+ size_t k = s - lcp - buf_;+ if (k + LEN + 4 > end_ || pat_->predict_match(&buf_[k + LEN])) {- set_current(loc);+ set_current(k); return true; } }@@ -2858,12 +2814,12 @@ bool Matcher::advance_chars_pma(size_t loc) loc = s - lcp - buf_; set_current_and_peek_more(loc); loc = cur_;- if (loc + LEN + min > end_)+ if (loc + LEN + min > end_ && eof_) return false; } } -/// Few chars followed by 4 to 8 characters+/// Few (2 or 3) chars followed by 4 to 8 characters template<uint8_t LEN> bool Matcher::advance_chars_pmh(size_t loc) {@@ -2871,11 +2827,11 @@ bool Matcher::advance_chars_pmh(size_t loc) const uint16_t lcs = LEN - 1; const char *chr = pat_->chr_; const size_t min = pat_->min_;-#if defined(HAVE_AVX512BW) || defined(HAVE_AVX2) || defined(HAVE_SSE2) while (true) { const char *s = buf_ + loc + lcp; const char *e = buf_ + end_ + lcp - LEN - min + 1;+#if defined(HAVE_AVX512BW) || defined(HAVE_AVX2) || defined(HAVE_SSE2) __m128i vlcp = _mm_set1_epi8(chr[lcp]); __m128i vlcs = _mm_set1_epi8(chr[lcs]); while (s <= e - 16)@@ -2891,10 +2847,10 @@ bool Matcher::advance_chars_pmh(size_t loc) if (LEN == 2 || (LEN == 3 ? s[offset + 1 - lcp] == chr[1] : std::memcmp(s + 1 - lcp + offset, chr + 1, LEN - 1) == 0)) {- loc = s - lcp + offset - buf_;- if (pat_->predict_match(&buf_[loc + LEN], min))+ size_t k = s - lcp + offset - buf_;+ if (pat_->predict_match(&buf_[k + LEN], min)) {- set_current(loc);+ set_current(k); return true; } }@@ -2902,19 +2858,7 @@ bool Matcher::advance_chars_pmh(size_t loc) } s += 16; }- loc = s - lcp - buf_;- set_current_and_peek_more(loc);- loc = cur_;- if (loc + LEN + min > end_)- return false;- if (loc + LEN + min + 15 > end_)- break;- } #elif defined(HAVE_NEON)- while (true)- {- const char *s = buf_ + loc + lcp;- const char *e = buf_ + end_ + lcp - LEN - min + 1; uint8x16_t vlcp = vdupq_n_u8(chr[lcp]); uint8x16_t vlcs = vdupq_n_u8(chr[lcs]); while (s <= e - 16)@@ -2928,68 +2872,56 @@ bool Matcher::advance_chars_pmh(size_t loc) uint64_t mask = vgetq_lane_u64(vmask64, 0); if (mask != 0) {- loc = s - lcp - buf_;+ size_t k = s - lcp - buf_; if (static_cast<uint32_t>(mask) == 0) { mask >>= 32;- loc += 4;+ k += 4; } do { if ((mask & 0xff) && (LEN == 2 ||- (LEN == 3 ? buf_[loc + 1] == chr[1] : std::memcmp(&buf_[loc + 1], chr + 1, LEN - 1) == 0)))+ (LEN == 3 ? buf_[k + 1] == chr[1] : std::memcmp(&buf_[k + 1], chr + 1, LEN - 1) == 0))) {- if (pat_->predict_match(&buf_[loc + LEN], min))+ if (pat_->predict_match(&buf_[k + LEN], min)) {- set_current(loc);+ set_current(k); return true; } } mask >>= 8;- ++loc;+ ++k; } while (mask != 0); } mask = vgetq_lane_u64(vmask64, 1); if (mask != 0) {- loc = s - lcp + 8 - buf_;+ size_t k = s - lcp + 8 - buf_; if (static_cast<uint32_t>(mask) == 0) { mask >>= 32;- loc += 4;+ k += 4; } do { if ((mask & 0xff) && (LEN == 2 ||- (LEN == 3 ? buf_[loc + 1] == chr[1] : std::memcmp(&buf_[loc + 1], chr + 1, LEN - 1) == 0)))+ (LEN == 3 ? buf_[k + 1] == chr[1] : std::memcmp(&buf_[k + 1], chr + 1, LEN - 1) == 0))) {- if (pat_->predict_match(&buf_[loc + LEN], min))+ if (pat_->predict_match(&buf_[k + LEN], min)) {- set_current(loc);+ set_current(k); return true; } } mask >>= 8;- ++loc;+ ++k; } while (mask != 0); } s += 16; }- loc = s - lcp - buf_;- set_current_and_peek_more(loc);- loc = cur_;- if (loc + LEN + min > end_)- return false;- if (loc + LEN + min + 15 > end_)- break;- } #endif- while (true)- {- const char *s = buf_ + loc + lcp;- const char *e = buf_ + end_ + lcp - LEN - min + 1; while (s < e) { do@@ -3003,10 +2935,10 @@ bool Matcher::advance_chars_pmh(size_t loc) if (LEN == 2 || (LEN == 3 ? s[1 - lcp] == chr[1] : std::memcmp(s + 1 - lcp, chr + 1, LEN - 1) == 0)) {- loc = s - lcp - buf_;- if (pat_->predict_match(&buf_[loc + LEN], min))+ size_t k = s - lcp - buf_;+ if (pat_->predict_match(&buf_[k + LEN], min)) {- set_current(loc);+ set_current(k); return true; } }@@ -3015,7 +2947,7 @@ bool Matcher::advance_chars_pmh(size_t loc) loc = s - lcp - buf_; set_current_and_peek_more(loc); loc = cur_;- if (loc + LEN + min > end_)+ if (loc + LEN + min > end_ && eof_) return false; } }@@ -3027,12 +2959,12 @@ bool Matcher::advance_string(size_t loc) const size_t len = pat_->len_; uint16_t lcp = pat_->lcp_; uint16_t lcs = pat_->lcs_;-#if defined(HAVE_AVX512BW) || defined(HAVE_AVX2) || defined(HAVE_SSE2)- // implements SSE2 string search scheme based on http://0x80.pl/articles/simd-friendly-karp-rabin.html while (true) { const char *s = buf_ + loc + lcp; const char *e = buf_ + end_ + lcp - len + 1;+#if defined(HAVE_AVX512BW) || defined(HAVE_AVX2) || defined(HAVE_SSE2)+ // implements SSE2 string search scheme based on http://0x80.pl/articles/simd-friendly-karp-rabin.html __m128i vlcp = _mm_set1_epi8(chr[lcp]); __m128i vlcs = _mm_set1_epi8(chr[lcs]); while (s <= e - 16)@@ -3047,42 +2979,19 @@ bool Matcher::advance_string(size_t loc) uint32_t offset = ctz(mask); if (std::memcmp(s - lcp + offset, chr, len) == 0) {- loc = s - lcp + offset - buf_;- set_current(loc);+ size_t k = s - lcp + offset - buf_;+ set_current(k); return true; } mask &= mask - 1; } s += 16; }- loc = s - lcp - buf_;- set_current_and_peek_more(loc);- loc = cur_;- if (loc + len > end_)- return false;- if (loc + len + 15 > end_)- break;- } #elif defined(HAVE_NEON)- while (true)- {- const char *s = buf_ + loc + lcp;- const char *e = buf_ + end_ + lcp - len + 1;+ // slightly faster when abstracted into a routine if (simd_advance_string_neon(s, e)) return true;- loc = s - lcp - buf_;- set_current_and_peek_more(loc);- loc = cur_;- if (loc + len > end_)- return false;- if (loc + len + 15 > end_)- break;- } #endif- while (true)- {- const char *s = buf_ + loc + lcp;- const char *e = buf_ + end_ + lcp - len + 1; while (s < e) { do@@ -3095,8 +3004,8 @@ bool Matcher::advance_string(size_t loc) } if (std::memcmp(s - lcp, chr, len) == 0) {- loc = s - lcp - buf_;- set_current(loc);+ size_t k = s - lcp - buf_;+ set_current(k); return true; } ++s;@@ -3104,7 +3013,7 @@ bool Matcher::advance_string(size_t loc) loc = s - lcp - buf_; set_current_and_peek_more(loc); loc = cur_;- if (loc + len > end_)+ if (loc + len > end_ && eof_) return false; } }@@ -3117,12 +3026,12 @@ bool Matcher::advance_string_pma(size_t loc) const size_t min = pat_->min_; const uint16_t lcp = pat_->lcp_; const uint16_t lcs = pat_->lcs_;-#if defined(HAVE_AVX512BW) || defined(HAVE_AVX2) || defined(HAVE_SSE2)- // implements SSE2 string search scheme based on http://0x80.pl/articles/simd-friendly-karp-rabin.html while (true) { const char *s = buf_ + loc + lcp; const char *e = buf_ + end_ + lcp - len - min + 1;+#if defined(HAVE_AVX512BW) || defined(HAVE_AVX2) || defined(HAVE_SSE2)+ // implements SSE2 string search scheme based on http://0x80.pl/articles/simd-friendly-karp-rabin.html __m128i vlcp = _mm_set1_epi8(chr[lcp]); __m128i vlcs = _mm_set1_epi8(chr[lcs]); while (s <= e - 16)@@ -3137,10 +3046,10 @@ bool Matcher::advance_string_pma(size_t loc) uint32_t offset = ctz(mask); if (std::memcmp(s - lcp + offset, chr, len) == 0) {- loc = s - lcp + offset - buf_;- if (loc + len + 4 > end_ || pat_->predict_match(&buf_[loc + len]))+ size_t k = s - lcp + offset - buf_;+ if (k + len + 4 > end_ || pat_->predict_match(&buf_[k + len])) {- set_current(loc);+ set_current(k); return true; } }@@ -3148,34 +3057,11 @@ bool Matcher::advance_string_pma(size_t loc) } s += 16; }- loc = s - lcp - buf_;- set_current_and_peek_more(loc);- loc = cur_;- if (loc + len + min > end_)- return false;- if (loc + len + min + 15 > end_)- break;- } #elif defined(HAVE_NEON)- while (true)- {- const char *s = buf_ + loc + lcp;- const char *e = buf_ + end_ + lcp - len - min + 1;+ // slightly faster when abstracted into a routine if (simd_advance_string_pma_neon(s, e)) return true;- loc = s - lcp - buf_;- set_current_and_peek_more(loc);- loc = cur_;- if (loc + len + min > end_)- return false;- if (loc + len + min + 15 > end_)- break;- } #endif- while (true)- {- const char *s = buf_ + loc + lcp;- const char *e = buf_ + end_ + lcp - len - min + 1; while (s < e) { do@@ -3188,10 +3074,10 @@ bool Matcher::advance_string_pma(size_t loc) } if (std::memcmp(s - lcp, chr, len) == 0) {- loc = s - lcp - buf_;- if (loc + len + 4 > end_ || pat_->predict_match(&buf_[loc + len]))+ size_t k = s - lcp - buf_;+ if (k + len + 4 > end_ || pat_->predict_match(&buf_[k + len])) {- set_current(loc);+ set_current(k); return true; } }@@ -3200,7 +3086,7 @@ bool Matcher::advance_string_pma(size_t loc) loc = s - lcp - buf_; set_current_and_peek_more(loc); loc = cur_;- if (loc + len + min > end_)+ if (loc + len + min > end_ && eof_) return false; } }@@ -3213,12 +3099,12 @@ bool Matcher::advance_string_pmh(size_t loc) const size_t min = pat_->min_; const uint16_t lcp = pat_->lcp_; const uint16_t lcs = pat_->lcs_;-#if defined(HAVE_AVX512BW) || defined(HAVE_AVX2) || defined(HAVE_SSE2)- // implements SSE2 string search scheme based on http://0x80.pl/articles/simd-friendly-karp-rabin.html while (true) { const char *s = buf_ + loc + lcp; const char *e = buf_ + end_ + lcp - len - min + 1;+#if defined(HAVE_AVX512BW) || defined(HAVE_AVX2) || defined(HAVE_SSE2)+ // implements SSE2 string search scheme based on http://0x80.pl/articles/simd-friendly-karp-rabin.html __m128i vlcp = _mm_set1_epi8(chr[lcp]); __m128i vlcs = _mm_set1_epi8(chr[lcs]); while (s <= e - 16)@@ -3233,10 +3119,10 @@ bool Matcher::advance_string_pmh(size_t loc) uint32_t offset = ctz(mask); if (std::memcmp(s - lcp + offset, chr, len) == 0) {- loc = s - lcp + offset - buf_;- if (pat_->predict_match(&buf_[loc + len], min))+ size_t k = s - lcp + offset - buf_;+ if (pat_->predict_match(&buf_[k + len], min)) {- set_current(loc);+ set_current(k); return true; } }@@ -3244,34 +3130,11 @@ bool Matcher::advance_string_pmh(size_t loc) } s += 16; }- loc = s - lcp - buf_;- set_current_and_peek_more(loc);- loc = cur_;- if (loc + len + min > end_)- return false;- if (loc + len + min + 15 > end_)- break;- } #elif defined(HAVE_NEON)- while (true)- {- const char *s = buf_ + loc + lcp;- const char *e = buf_ + end_ + lcp - len - min + 1;+ // slightly faster when abstracted into a routine if (simd_advance_string_pmh_neon(s, e)) return true;- loc = s - lcp - buf_;- set_current_and_peek_more(loc);- loc = cur_;- if (loc + len + min > end_)- return false;- if (loc + len + min + 15 > end_)- break;- } #endif- while (true)- {- const char *s = buf_ + loc + lcp;- const char *e = buf_ + end_ + lcp - len - min + 1; while (s < e) { do@@ -3284,10 +3147,10 @@ bool Matcher::advance_string_pmh(size_t loc) } if (std::memcmp(s - lcp, chr, len) == 0) {- loc = s - lcp - buf_;- if (pat_->predict_match(&buf_[loc + len], min))+ size_t k = s - lcp - buf_;+ if (pat_->predict_match(&buf_[k + len], min)) {- set_current(loc);+ set_current(k); return true; } }@@ -3296,7 +3159,7 @@ bool Matcher::advance_string_pmh(size_t loc) loc = s - lcp - buf_; set_current_and_peek_more(loc); loc = cur_;- if (loc + len + min > end_)+ if (loc + len + min > end_ && eof_) return false; } }@@ -3304,7 +3167,7 @@ bool Matcher::advance_string_pmh(size_t loc) #if defined(HAVE_NEON) // Implements NEON/AArch64 string search scheme based on http://0x80.pl/articles/simd-friendly-karp-rabin.html 64 bit optimized-bool Matcher::simd_advance_string_neon(const char *&s, const char *e)+bool Matcher::simd_advance_string_neon(const char *& s, const char *e) { const uint16_t lcp = pat_->lcp_; const uint16_t lcs = pat_->lcs_;@@ -3323,41 +3186,41 @@ bool Matcher::simd_advance_string_neon(const char *&s, const char *e) uint64_t mask = vgetq_lane_u64(vmask64, 0); if (mask != 0) {- size_t loc = s - lcp - buf_;+ size_t k = s - lcp - buf_; if (static_cast<uint32_t>(mask) == 0) { mask >>= 32;- loc += 4;+ k += 4; } do {- if ((mask & 0xff) && std::memcmp(&buf_[loc], chr, len) == 0)+ if ((mask & 0xff) && std::memcmp(&buf_[k], chr, len) == 0) {- set_current(loc);+ set_current(k); return true; } mask >>= 8;- ++loc;+ ++k; } while (mask != 0); } mask = vgetq_lane_u64(vmask64, 1); if (mask != 0) {- size_t loc = s - lcp + 8 - buf_;+ size_t k = s - lcp + 8 - buf_; if (static_cast<uint32_t>(mask) == 0) { mask >>= 32;- loc += 4;+ k += 4; } do {- if ((mask & 0xff) && std::memcmp(&buf_[loc], chr, len) == 0)+ if ((mask & 0xff) && std::memcmp(&buf_[k], chr, len) == 0) {- set_current(loc);+ set_current(k); return true; } mask >>= 8;- ++loc;+ ++k; } while (mask != 0); } s += 16;@@ -3366,7 +3229,7 @@ bool Matcher::simd_advance_string_neon(const char *&s, const char *e) } // Implements NEON/AArch64 string search scheme based on http://0x80.pl/articles/simd-friendly-karp-rabin.html 64 bit optimized-bool Matcher::simd_advance_string_pma_neon(const char *&s, const char *e)+bool Matcher::simd_advance_string_pma_neon(const char *& s, const char *e) { const uint16_t lcp = pat_->lcp_; const uint16_t lcs = pat_->lcs_;@@ -3385,47 +3248,47 @@ bool Matcher::simd_advance_string_pma_neon(const char *&s, const char *e) uint64_t mask = vgetq_lane_u64(vmask64, 0); if (mask != 0) {- size_t loc = s - lcp - buf_;+ size_t k = s - lcp - buf_; if (static_cast<uint32_t>(mask) == 0) { mask >>= 32;- loc += 4;+ k += 4; } do {- if ((mask & 0xff) && std::memcmp(&buf_[loc], chr, len) == 0)+ if ((mask & 0xff) && std::memcmp(&buf_[k], chr, len) == 0) {- if (loc + len + 4 > end_ || pat_->predict_match(&buf_[loc + len]))+ if (k + len + 4 > end_ || pat_->predict_match(&buf_[k + len])) {- set_current(loc);+ set_current(k); return true; } } mask >>= 8;- ++loc;+ ++k; } while (mask != 0); } mask = vgetq_lane_u64(vmask64, 1); if (mask != 0) {- size_t loc = s - lcp + 8 - buf_;+ size_t k = s - lcp + 8 - buf_; if (static_cast<uint32_t>(mask) == 0) { mask >>= 32;- loc += 4;+ k += 4; } do {- if ((mask & 0xff) && std::memcmp(&buf_[loc], chr, len) == 0)+ if ((mask & 0xff) && std::memcmp(&buf_[k], chr, len) == 0) {- if (loc + len + 4 > end_ || pat_->predict_match(&buf_[loc + len]))+ if (k + len + 4 > end_ || pat_->predict_match(&buf_[k + len])) {- set_current(loc);+ set_current(k); return true; } } mask >>= 8;- ++loc;+ ++k; } while (mask != 0); } s += 16;@@ -3434,7 +3297,7 @@ bool Matcher::simd_advance_string_pma_neon(const char *&s, const char *e) } // Implements NEON/AArch64 string search scheme based on http://0x80.pl/articles/simd-friendly-karp-rabin.html 64 bit optimized-bool Matcher::simd_advance_string_pmh_neon(const char *&s, const char *e)+bool Matcher::simd_advance_string_pmh_neon(const char *& s, const char *e) { const uint16_t lcp = pat_->lcp_; const uint16_t lcs = pat_->lcs_;@@ -3454,47 +3317,47 @@ bool Matcher::simd_advance_string_pmh_neon(const char *&s, const char *e) uint64_t mask = vgetq_lane_u64(vmask64, 0); if (mask != 0) {- size_t loc = s - lcp - buf_;+ size_t k = s - lcp - buf_; if (static_cast<uint32_t>(mask) == 0) { mask >>= 32;- loc += 4;+ k += 4; } do {- if ((mask & 0xff) && std::memcmp(&buf_[loc], chr, len) == 0)+ if ((mask & 0xff) && std::memcmp(&buf_[k], chr, len) == 0) {- if (pat_->predict_match(&buf_[loc + len], min))+ if (pat_->predict_match(&buf_[k + len], min)) {- set_current(loc);+ set_current(k); return true; } } mask >>= 8;- ++loc;+ ++k; } while (mask != 0); } mask = vgetq_lane_u64(vmask64, 1); if (mask != 0) {- size_t loc = s - lcp + 8 - buf_;+ size_t k = s - lcp + 8 - buf_; if (static_cast<uint32_t>(mask) == 0) { mask >>= 32;- loc += 4;+ k += 4; } do {- if ((mask & 0xff) && std::memcmp(&buf_[loc], chr, len) == 0)+ if ((mask & 0xff) && std::memcmp(&buf_[k], chr, len) == 0) {- if (pat_->predict_match(&buf_[loc + len], min))+ if (pat_->predict_match(&buf_[k + len], min)) {- set_current(loc);+ set_current(k); return true; } } mask >>= 8;- ++loc;+ ++k; } while (mask != 0); } s += 16;@@ -3534,8 +3397,8 @@ bool Matcher::advance_string_bm(size_t loc) } if (p < chr) {- loc = q - buf_ + 1;- set_current(loc);+ k = q - buf_ + 1;+ set_current(k); return true; } if (chr + bmd >= p)@@ -3555,7 +3418,7 @@ bool Matcher::advance_string_bm(size_t loc) loc = s - buf_; set_current_and_peek_more(loc); loc = cur_;- if (loc + len > end_)+ if (loc + len > end_ && eof_) return false; } }@@ -3590,10 +3453,10 @@ bool Matcher::advance_string_bm_pma(size_t loc) } if (p < chr) {- loc = q - buf_ + 1;- if (loc + len + 4 > end_ || pat_->predict_match(&buf_[loc + len]))+ k = q - buf_ + 1;+ if (k + len + 4 > end_ || pat_->predict_match(&buf_[k + len])) {- set_current(loc);+ set_current(k); return true; } }@@ -3614,7 +3477,7 @@ bool Matcher::advance_string_bm_pma(size_t loc) loc = s - buf_; set_current_and_peek_more(loc); loc = cur_;- if (loc + len > end_)+ if (loc + len > end_ && eof_) return false; } }@@ -3650,10 +3513,10 @@ bool Matcher::advance_string_bm_pmh(size_t loc) } if (p < chr) {- loc = q - buf_ + 1;- if (loc + len + min > end_ || pat_->predict_match(&buf_[loc + len], min))+ k = q - buf_ + 1;+ if (k + len + min > end_ || pat_->predict_match(&buf_[k + len], min)) {- set_current(loc);+ set_current(k); return true; } }@@ -3674,7 +3537,7 @@ bool Matcher::advance_string_bm_pmh(size_t loc) loc = s - buf_; set_current_and_peek_more(loc); loc = cur_;- if (loc + len > end_)+ if (loc + len > end_ && eof_) return false; } }
@@ -30,7 +30,7 @@ @file matcher_avx2.cpp @brief RE/flex matcher engine @author Robert van Engelen - engelen@genivia.com-@copyright (c) 2016-2024, Robert van Engelen, Genivia Inc. All rights reserved.+@copyright (c) 2016-2025, Robert van Engelen, Genivia Inc. All rights reserved. @copyright (c) BSD-3 License - see LICENSE.txt */ @@ -192,8 +192,10 @@ bool Matcher::simd_advance_pattern_pin1_pma_avx2(size_t loc) const size_t min = pat_->min_; const uint16_t lcp = pat_->lcp_; const uint16_t lcs = pat_->lcs_;- __m256i vlcp = _mm256_set1_epi8(chr[0]);- __m256i vlcs = _mm256_set1_epi8(chr[1]);+ const char chr0 = chr[0];+ const char chr1 = chr[1];+ __m256i vlcp = _mm256_set1_epi8(chr0);+ __m256i vlcs = _mm256_set1_epi8(chr1); while (true) { const char *s = buf_ + loc + lcp;@@ -208,23 +210,35 @@ bool Matcher::simd_advance_pattern_pin1_pma_avx2(size_t loc) while (mask != 0) { uint32_t offset = ctz(mask);- loc = s - lcp + offset - buf_;- set_current(loc);- if (loc + 4 > end_ || pat_->predict_match(&buf_[loc]))+ size_t k = s - lcp + offset - buf_;+ set_current(k);+ if (k + 4 > end_ || pat_->predict_match(&buf_[k])) return true; mask &= mask - 1; } s += 32; }- loc = s - lcp - buf_;- set_current_and_peek_more(loc);- loc = cur_;- if (loc + min > end_)- return false;- if (loc + min + 31 > end_)- break;+ e = buf_ + end_;+ if (s < e && (s = static_cast<const char*>(std::memchr(s, chr0, e - s))) != NULL)+ {+ s -= lcp;+ loc = s - buf_;+ if (s > e - 4 || (s[lcs] == chr1 && pat_->predict_match(s)))+ {+ set_current(loc);+ return true;+ }+ ++loc;+ }+ else+ {+ loc = std::max<size_t>(loc, e - lcp - buf_);+ set_current_and_peek_more(loc);+ loc = cur_;+ if (loc + min > end_ && eof_)+ return false;+ } }- return advance_pattern_pin1_pma(loc); } // My homegrown "needle search" method when needle pin=1@@ -234,8 +248,10 @@ bool Matcher::simd_advance_pattern_pin1_pmh_avx2(size_t loc) const char *chr = pat_->chr_; const uint16_t lcp = pat_->lcp_; const uint16_t lcs = pat_->lcs_;- __m256i vlcp = _mm256_set1_epi8(chr[0]);- __m256i vlcs = _mm256_set1_epi8(chr[1]);+ const char chr0 = chr[0];+ const char chr1 = chr[1];+ __m256i vlcp = _mm256_set1_epi8(chr0);+ __m256i vlcs = _mm256_set1_epi8(chr1); while (true) { const char *s = buf_ + loc + lcp;@@ -250,23 +266,35 @@ bool Matcher::simd_advance_pattern_pin1_pmh_avx2(size_t loc) while (mask != 0) { uint32_t offset = ctz(mask);- loc = s - lcp + offset - buf_;- set_current(loc);- if (pat_->predict_match(&buf_[loc], MIN))+ size_t k = s - lcp + offset - buf_;+ set_current(k);+ if (pat_->predict_match(&buf_[k], MIN)) return true; mask &= mask - 1; } s += 32; }- loc = s - lcp - buf_;- set_current_and_peek_more(loc);- loc = cur_;- if (loc + MIN > end_)- return false;- if (loc + MIN + 31 > end_)- break;+ e = buf_ + end_;+ if (s < e && (s = static_cast<const char*>(std::memchr(s, chr0, e - s))) != NULL)+ {+ s -= lcp;+ loc = s - buf_;+ if (s + MIN > e || (s[lcs] == chr1 && pat_->predict_match(s, MIN)))+ {+ set_current(loc);+ return true;+ }+ ++loc;+ }+ else+ {+ loc = std::max<size_t>(loc, e - lcp - buf_);+ set_current_and_peek_more(loc);+ loc = cur_;+ if (loc + MIN > end_ && eof_)+ return false;+ } }- return advance_pattern_pin1_pmh<MIN>(loc); } // My homegrown "needle search" methods@@ -288,25 +316,31 @@ bool Matcher::simd_advance_pattern_pin##N##_one_avx2(size_t loc) \ while (mask != 0) \ { \ uint32_t offset = ctz(mask); \- loc = s + offset - buf_; \- if (loc + 4 > end_ || pat_->predict_match(&buf_[loc])) \+ size_t k = s + offset - buf_; \+ if (k + 4 > end_ || pat_->predict_match(&buf_[k])) \ { \- set_current(loc); \+ set_current(k); \ return true; \ } \ mask &= mask - 1; \ } \ s += 32; \ } \+ while (s < e - 3) \+ { \+ if (pat_->predict_match(s++)) \+ { \+ size_t k = s - buf_ - 1; \+ set_current(k); \+ return true; \+ } \+ } \ loc = s - buf_; \ set_current_and_peek_more(loc); \ loc = cur_; \- if (loc + 1 > end_) \- return false; \- if (loc + 32 > end_) \- break; \+ if (loc + 3 >= end_) \+ return true; \ } \- return advance_pattern_pma(loc); \ } ADV_PAT_PIN_ONE(2, \@@ -459,25 +493,35 @@ bool Matcher::simd_advance_pattern_pin##N##_pma_avx2(size_t loc) \ while (mask != 0) \ { \ uint32_t offset = ctz(mask); \- loc = s - lcp + offset - buf_; \- if (loc + 4 > end_ || pat_->predict_match(&buf_[loc])) \+ size_t k = s - lcp + offset - buf_; \+ if (k + 4 > end_ || pat_->predict_match(&buf_[k])) \ { \- set_current(loc); \+ set_current(k); \ return true; \ } \ mask &= mask - 1; \ } \ s += 32; \ } \- loc = s - lcp - buf_; \+ s -= lcp; \+ e = buf_ + end_ - 3; \+ while (s < e) \+ { \+ if (pat_->predict_match(s++)) \+ { \+ size_t k = s - buf_ - 1; \+ set_current(k); \+ return true; \+ } \+ } \+ loc = s - buf_; \ set_current_and_peek_more(loc); \ loc = cur_; \- if (loc + min > end_) \+ if (loc + min > end_ && eof_) \ return false; \- if (loc + min + 31 > end_) \- break; \+ if (loc + 3 >= end_) \+ return true; \ } \- return advance_pattern_pma(loc); \ } \ \ template <uint8_t MIN> \@@ -502,25 +546,33 @@ bool Matcher::simd_advance_pattern_pin##N##_pmh_avx2(size_t loc) \ while (mask != 0) \ { \ uint32_t offset = ctz(mask); \- loc = s - lcp + offset - buf_; \- if (pat_->predict_match(&buf_[loc], MIN)) \+ size_t k = s - lcp + offset - buf_; \+ if (pat_->predict_match(&buf_[k], MIN)) \ { \- set_current(loc); \+ set_current(k); \ return true; \ } \ mask &= mask - 1; \ } \ s += 32; \ } \- loc = s - lcp - buf_; \+ s -= lcp; \+ e = buf_ + end_ - MIN + 1; \+ while (s < e) \+ { \+ if (pat_->predict_match(s++, MIN)) \+ { \+ size_t k = s - buf_ - 1; \+ set_current(k); \+ return true; \+ } \+ } \+ loc = s - buf_; \ set_current_and_peek_more(loc); \ loc = cur_; \- if (loc + MIN > end_) \+ if (loc + MIN > end_ && eof_) \ return false; \- if (loc + MIN + 31 > end_) \- break; \ } \- return advance_pattern_min4<MIN>(loc); \ } ADV_PAT_PIN(2, \@@ -785,26 +837,26 @@ bool Matcher::simd_advance_pattern_min4_avx2(size_t loc) uint32_t mask = _mm_extract_epi32(_mm_shuffle_epi8(vstate, vselect), 0); if ((mask & 0x00000008) == 0 && pat_->predict_match(s - MIN + 0, MIN)) {- loc = s - buf_ - MIN + 0;- set_current(loc);+ size_t k = s - buf_ - MIN + 0;+ set_current(k); return true; } if ((mask & 0x00000404) == 0 && pat_->predict_match(s - MIN + 1, MIN)) {- loc = s - buf_ - MIN + 1;- set_current(loc);+ size_t k = s - buf_ - MIN + 1;+ set_current(k); return true; } if ((mask & 0x00020202) == 0 && pat_->predict_match(s - MIN + 2, MIN)) {- loc = s - buf_ - MIN + 2;- set_current(loc);+ size_t k = s - buf_ - MIN + 2;+ set_current(k); return true; } if ((mask & 0x01010101) == 0 && pat_->predict_match(s - MIN + 3, MIN)) {- loc = s - buf_ - MIN + 3;- set_current(loc);+ size_t k = s - buf_ - MIN + 3;+ set_current(k); return true; } // butterfly-or:@@ -853,23 +905,39 @@ bool Matcher::simd_advance_chars_avx2(size_t loc) if (LEN == 2 || (LEN == 3 ? s[offset + 1 - lcp] == chr[1] : std::memcmp(s + 1 - lcp + offset, chr + 1, LEN - 2) == 0)) {- loc = s - lcp + offset - buf_;- set_current(loc);+ size_t k = s - lcp + offset - buf_;+ set_current(k); return true; } mask &= mask - 1; } s += 32; }+ while (s < e)+ {+ do+ s = static_cast<const char*>(std::memchr(s, chr[lcp], e - s));+ while (s != NULL && s[lcs - lcp] != chr[lcs] && ++s < e);+ if (s == NULL || s >= e)+ {+ s = e;+ break;+ }+ if (LEN == 2 ||+ (LEN == 3 ? s[1 - lcp] == chr[1] : std::memcmp(s + 1 - lcp, chr + 1, LEN - 1) == 0))+ {+ size_t k = s - lcp - buf_;+ set_current(k);+ return true;+ }+ ++s;+ } loc = s - lcp - buf_; set_current_and_peek_more(loc); loc = cur_;- if (loc + LEN > end_)+ if (loc + LEN > end_ && eof_) return false;- if (loc + LEN + 31 > end_)- break; }- return advance_chars<LEN>(loc); } /// Few chars followed by 2 to 3 minimal char pattern@@ -883,7 +951,7 @@ bool Matcher::simd_advance_chars_pma_avx2(size_t loc) while (true) { const char *s = buf_ + loc + lcp;- const char *e = buf_ + end_ + lcp - LEN + 1;+ const char *e = buf_ + end_ + lcp - LEN - min + 1; __m256i vlcp = _mm256_set1_epi8(chr[lcp]); __m256i vlcs = _mm256_set1_epi8(chr[lcs]); while (s <= e - 32)@@ -899,10 +967,10 @@ bool Matcher::simd_advance_chars_pma_avx2(size_t loc) if (LEN == 2 || (LEN == 3 ? s[offset + 1 - lcp] : std::memcmp(s + 1 - lcp + offset, chr + 1, LEN - 2) == 0)) {- loc = s - lcp + offset - buf_;- if (loc + LEN + 4 > end_ || pat_->predict_match(&buf_[loc + LEN]))+ size_t k = s - lcp + offset - buf_;+ if (k + LEN + 4 > end_ || pat_->predict_match(&buf_[k + LEN])) {- set_current(loc);+ set_current(k); return true; } }@@ -910,15 +978,34 @@ bool Matcher::simd_advance_chars_pma_avx2(size_t loc) } s += 32; }+ while (s < e)+ {+ do+ s = static_cast<const char*>(std::memchr(s, chr[lcp], e - s));+ while (s != NULL && s[lcs - lcp] != chr[lcs] && ++s < e);+ if (s == NULL || s >= e)+ {+ s = e;+ break;+ }+ if (LEN == 2 ||+ (LEN == 3 ? s[1 - lcp] == chr[1] : std::memcmp(s + 1 - lcp, chr + 1, LEN - 1) == 0))+ {+ size_t k = s - lcp - buf_;+ if (k + LEN + 4 > end_ || pat_->predict_match(&buf_[k + LEN]))+ {+ set_current(k);+ return true;+ }+ }+ ++s;+ } loc = s - lcp - buf_; set_current_and_peek_more(loc); loc = cur_;- if (loc + LEN + min > end_)+ if (loc + LEN + min > end_ && eof_) return false;- if (loc + LEN + min + 31 > end_)- break; }- return advance_chars_pma<LEN>(loc); } /// Few chars followed by 4 minimal char pattern@@ -932,7 +1019,7 @@ bool Matcher::simd_advance_chars_pmh_avx2(size_t loc) while (true) { const char *s = buf_ + loc + lcp;- const char *e = buf_ + end_ + lcp - LEN + 1;+ const char *e = buf_ + end_ + lcp - LEN - min + 1; __m256i vlcp = _mm256_set1_epi8(chr[lcp]); __m256i vlcs = _mm256_set1_epi8(chr[lcs]); while (s <= e - 32)@@ -948,24 +1035,43 @@ bool Matcher::simd_advance_chars_pmh_avx2(size_t loc) if (LEN == 2 || (LEN == 3 ? s[offset + 1 - lcp] == chr[1] : std::memcmp(s + 1 - lcp + offset, chr + 1, LEN - 2) == 0)) {- loc = s - lcp + offset - buf_;- set_current(loc);- if (loc + LEN + min > end_ || pat_->predict_match(&buf_[loc + LEN], min))+ size_t k = s - lcp + offset - buf_;+ set_current(k);+ if (k + LEN + min > end_ || pat_->predict_match(&buf_[k + LEN], min)) return true; } mask &= mask - 1; } s += 32; }+ while (s < e)+ {+ do+ s = static_cast<const char*>(std::memchr(s, chr[lcp], e - s));+ while (s != NULL && s[lcs - lcp] != chr[lcs] && ++s < e);+ if (s == NULL || s >= e)+ {+ s = e;+ break;+ }+ if (LEN == 2 ||+ (LEN == 3 ? s[1 - lcp] == chr[1] : std::memcmp(s + 1 - lcp, chr + 1, LEN - 1) == 0))+ {+ size_t k = s - lcp - buf_;+ if (pat_->predict_match(&buf_[k + LEN], min))+ {+ set_current(k);+ return true;+ }+ }+ ++s;+ } loc = s - lcp - buf_; set_current_and_peek_more(loc); loc = cur_;- if (loc + LEN + min > end_)+ if (loc + LEN + min > end_ && eof_) return false;- if (loc + LEN + min + 31 > end_)- break; }- return advance_chars_pmh<LEN>(loc); } /// Implements AVX2 string search scheme based on http://0x80.pl/articles/simd-friendly-karp-rabin.html@@ -993,23 +1099,38 @@ bool Matcher::simd_advance_string_avx2(size_t loc) uint32_t offset = ctz(mask); if (std::memcmp(s - lcp + offset, chr, len) == 0) {- loc = s - lcp + offset - buf_;- set_current(loc);+ size_t k = s - lcp + offset - buf_;+ set_current(k); return true; } mask &= mask - 1; } s += 32; }+ while (s < e)+ {+ do+ s = static_cast<const char*>(std::memchr(s, chr[lcp], e - s));+ while (s != NULL && s[lcs - lcp] != chr[lcs] && ++s < e);+ if (s == NULL || s >= e)+ {+ s = e;+ break;+ }+ if (std::memcmp(s - lcp, chr, len) == 0)+ {+ size_t k = s - lcp - buf_;+ set_current(k);+ return true;+ }+ ++s;+ } loc = s - lcp - buf_; set_current_and_peek_more(loc); loc = cur_;- if (loc + len > end_)+ if (loc + len > end_ && eof_) return false;- if (loc + len + 31 > end_)- break; }- return advance_string(loc); } /// Implements AVX2 string search scheme based on http://0x80.pl/articles/simd-friendly-karp-rabin.html@@ -1023,7 +1144,7 @@ bool Matcher::simd_advance_string_pma_avx2(size_t loc) while (true) { const char *s = buf_ + loc + lcp;- const char *e = buf_ + end_ + lcp - len + 1;+ const char *e = buf_ + end_ + lcp - len - min + 1; __m256i vlcp = _mm256_set1_epi8(chr[lcp]); __m256i vlcs = _mm256_set1_epi8(chr[lcs]); while (s <= e - 32)@@ -1038,10 +1159,10 @@ bool Matcher::simd_advance_string_pma_avx2(size_t loc) uint32_t offset = ctz(mask); if (std::memcmp(s - lcp + offset, chr, len) == 0) {- loc = s - lcp + offset - buf_;- if (loc + len + 4 > end_ || pat_->predict_match(&buf_[loc + len]))+ size_t k = s - lcp + offset - buf_;+ if (k + len + 4 > end_ || pat_->predict_match(&buf_[k + len])) {- set_current(loc);+ set_current(k); return true; } }@@ -1049,15 +1170,33 @@ bool Matcher::simd_advance_string_pma_avx2(size_t loc) } s += 32; }+ while (s < e)+ {+ do+ s = static_cast<const char*>(std::memchr(s, chr[lcp], e - s));+ while (s != NULL && s[lcs - lcp] != chr[lcs] && ++s < e);+ if (s == NULL || s >= e)+ {+ s = e;+ break;+ }+ if (std::memcmp(s - lcp, chr, len) == 0)+ {+ size_t k = s - lcp - buf_;+ if (k + len + 4 > end_ || pat_->predict_match(&buf_[k + len]))+ {+ set_current(k);+ return true;+ }+ }+ ++s;+ } loc = s - lcp - buf_; set_current_and_peek_more(loc); loc = cur_;- if (loc + len + min > end_)+ if (loc + len + min > end_ && eof_) return false;- if (loc + len + min + 31 > end_)- break; }- return advance_string_pma(loc); } /// Implements AVX2 string search scheme based on http://0x80.pl/articles/simd-friendly-karp-rabin.html@@ -1071,7 +1210,7 @@ bool Matcher::simd_advance_string_pmh_avx2(size_t loc) while (true) { const char *s = buf_ + loc + lcp;- const char *e = buf_ + end_ + lcp - len + 1;+ const char *e = buf_ + end_ + lcp - len - min + 1; __m256i vlcp = _mm256_set1_epi8(chr[lcp]); __m256i vlcs = _mm256_set1_epi8(chr[lcs]); while (s <= e - 32)@@ -1086,24 +1225,42 @@ bool Matcher::simd_advance_string_pmh_avx2(size_t loc) uint32_t offset = ctz(mask); if (std::memcmp(s - lcp + offset, chr, len) == 0) {- loc = s - lcp + offset - buf_;- set_current(loc);- if (loc + len + min > end_ || pat_->predict_match(&buf_[loc + len], min))+ size_t k = s - lcp + offset - buf_;+ set_current(k);+ if (k + len + min > end_ || pat_->predict_match(&buf_[k + len], min)) return true; } mask &= mask - 1; } s += 32; }+ while (s < e)+ {+ do+ s = static_cast<const char*>(std::memchr(s, chr[lcp], e - s));+ while (s != NULL && s[lcs - lcp] != chr[lcs] && ++s < e);+ if (s == NULL || s >= e)+ {+ s = e;+ break;+ }+ if (std::memcmp(s - lcp, chr, len) == 0)+ {+ size_t k = s - lcp - buf_;+ if (pat_->predict_match(&buf_[k + len], min))+ {+ set_current(k);+ return true;+ }+ }+ ++s;+ } loc = s - lcp - buf_; set_current_and_peek_more(loc); loc = cur_;- if (loc + len + min > end_)+ if (loc + len + min > end_ && eof_) return false;- if (loc + len + min + 31 > end_)- break; }- return advance_string_pmh(loc); } #else
@@ -30,7 +30,7 @@ @file matcher_avx512bw.cpp @brief RE/flex matcher engine @author Robert van Engelen - engelen@genivia.com-@copyright (c) 2016-2024, Robert van Engelen, Genivia Inc. All rights reserved.+@copyright (c) 2016-2025, Robert van Engelen, Genivia Inc. All rights reserved. @copyright (c) BSD-3 License - see LICENSE.txt */ @@ -110,24 +110,39 @@ bool Matcher::simd_advance_chars_avx512bw(size_t loc) if (LEN == 2 || (LEN == 3 ? s[offset + 1 - lcp] == chr[1] : std::memcmp(s + 1 - lcp + offset, chr + 1, LEN - 2) == 0)) {- loc = s - lcp + offset - buf_;- set_current(loc);+ size_t k = s - lcp + offset - buf_;+ set_current(k); return true; } mask &= mask - 1; } s += 64; }- s -= lcp;- loc = s - buf_;+ while (s < e)+ {+ do+ s = static_cast<const char*>(std::memchr(s, chr[lcp], e - s));+ while (s != NULL && s[lcs - lcp] != chr[lcs] && ++s < e);+ if (s == NULL || s >= e)+ {+ s = e;+ break;+ }+ if (LEN == 2 ||+ (LEN == 3 ? s[1 - lcp] == chr[1] : std::memcmp(s + 1 - lcp, chr + 1, LEN - 1) == 0))+ {+ size_t k = s - lcp - buf_;+ set_current(k);+ return true;+ }+ ++s;+ }+ loc = s - lcp - buf_; set_current_and_peek_more(loc); loc = cur_;- if (loc + LEN > end_)+ if (loc + LEN > end_ && eof_) return false;- if (loc + LEN + 63 > end_)- break; }- return advance_chars<LEN>(loc); } /// Few chars followed by 2 to 3 minimal char pattern@@ -141,7 +156,7 @@ bool Matcher::simd_advance_chars_pma_avx512bw(size_t loc) while (true) { const char *s = buf_ + loc + lcp;- const char *e = buf_ + end_ + lcp - LEN + 1;+ const char *e = buf_ + end_ + lcp - LEN - min + 1; __m512i vlcp = _mm512_set1_epi8(chr[lcp]); __m512i vlcs = _mm512_set1_epi8(chr[lcs]); while (s <= e - 64)@@ -155,10 +170,10 @@ bool Matcher::simd_advance_chars_pma_avx512bw(size_t loc) if (LEN == 2 || (LEN == 3 ? s[offset + 1 - lcp] == chr[1] : std::memcmp(s + 1 - lcp + offset, chr + 1, LEN - 2) == 0)) {- loc = s - lcp + offset - buf_;- if (loc + LEN + 4 > end_ || pat_->predict_match(&buf_[loc + LEN]))+ size_t k = s - lcp + offset - buf_;+ if (k + LEN + 4 > end_ || pat_->predict_match(&buf_[k + LEN])) {- set_current(loc);+ set_current(k); return true; } }@@ -166,16 +181,34 @@ bool Matcher::simd_advance_chars_pma_avx512bw(size_t loc) } s += 64; }- s -= lcp;- loc = s - buf_;+ while (s < e)+ {+ do+ s = static_cast<const char*>(std::memchr(s, chr[lcp], e - s));+ while (s != NULL && s[lcs - lcp] != chr[lcs] && ++s < e);+ if (s == NULL || s >= e)+ {+ s = e;+ break;+ }+ if (LEN == 2 ||+ (LEN == 3 ? s[1 - lcp] == chr[1] : std::memcmp(s + 1 - lcp, chr + 1, LEN - 1) == 0))+ {+ size_t k = s - lcp - buf_;+ if (k + LEN + 4 > end_ || pat_->predict_match(&buf_[k + LEN]))+ {+ set_current(k);+ return true;+ }+ }+ ++s;+ }+ loc = s - lcp - buf_; set_current_and_peek_more(loc); loc = cur_;- if (loc + LEN + min > end_)+ if (loc + LEN + min > end_ && eof_) return false;- if (loc + LEN + min + 63 > end_)- break; }- return advance_chars_pma<LEN>(loc); } /// Few chars followed by 4 minimal char pattern@@ -189,7 +222,7 @@ bool Matcher::simd_advance_chars_pmh_avx512bw(size_t loc) while (true) { const char *s = buf_ + loc + lcp;- const char *e = buf_ + end_ + lcp - LEN + 1;+ const char *e = buf_ + end_ + lcp - LEN - min + 1; __m512i vlcp = _mm512_set1_epi8(chr[lcp]); __m512i vlcs = _mm512_set1_epi8(chr[lcs]); while (s <= e - 64)@@ -203,10 +236,10 @@ bool Matcher::simd_advance_chars_pmh_avx512bw(size_t loc) if (LEN == 2 || (LEN == 3 ? s[offset + 1 - lcp] == chr[1] : std::memcmp(s + 1 - lcp + offset, chr + 1, LEN - 2) == 0)) {- loc = s - lcp + offset - buf_;- if (loc + LEN + min > end_ || pat_->predict_match(&buf_[loc + LEN], min))+ size_t k = s - lcp + offset - buf_;+ if (k + LEN + min > end_ || pat_->predict_match(&buf_[k + LEN], min)) {- set_current(loc);+ set_current(k); return true; } }@@ -214,16 +247,34 @@ bool Matcher::simd_advance_chars_pmh_avx512bw(size_t loc) } s += 64; }- s -= lcp;- loc = s - buf_;+ while (s < e)+ {+ do+ s = static_cast<const char*>(std::memchr(s, chr[lcp], e - s));+ while (s != NULL && s[lcs - lcp] != chr[lcs] && ++s < e);+ if (s == NULL || s >= e)+ {+ s = e;+ break;+ }+ if (LEN == 2 ||+ (LEN == 3 ? s[1 - lcp] == chr[1] : std::memcmp(s + 1 - lcp, chr + 1, LEN - 1) == 0))+ {+ size_t k = s - lcp - buf_;+ if (pat_->predict_match(&buf_[k + LEN], min))+ {+ set_current(k);+ return true;+ }+ }+ ++s;+ }+ loc = s - lcp - buf_; set_current_and_peek_more(loc); loc = cur_;- if (loc + LEN + min > end_)+ if (loc + LEN + min > end_ && eof_) return false;- if (loc + LEN + min + 63 > end_)- break; }- return advance_chars_pmh<LEN>(loc); } /// Implements AVX512BW string search scheme based on http://0x80.pl/articles/simd-friendly-karp-rabin.html@@ -249,24 +300,38 @@ bool Matcher::simd_advance_string_avx512bw(size_t loc) uint32_t offset = ctzl(mask); if (std::memcmp(s - lcp + offset, chr, len) == 0) {- loc = s - lcp + offset - buf_;- set_current(loc);+ size_t k = s - lcp + offset - buf_;+ set_current(k); return true; } mask &= mask - 1; } s += 64; }- s -= lcp;- loc = s - buf_;+ while (s < e)+ {+ do+ s = static_cast<const char*>(std::memchr(s, chr[lcp], e - s));+ while (s != NULL && s[lcs - lcp] != chr[lcs] && ++s < e);+ if (s == NULL || s >= e)+ {+ s = e;+ break;+ }+ if (std::memcmp(s - lcp, chr, len) == 0)+ {+ size_t k = s - lcp - buf_;+ set_current(k);+ return true;+ }+ ++s;+ }+ loc = s - lcp - buf_; set_current_and_peek_more(loc); loc = cur_;- if (loc + len > end_)+ if (loc + len > end_ && eof_) return false;- if (loc + len + 63 > end_)- break; }- return advance_string(loc); } /// Implements AVX512BW string search scheme based on http://0x80.pl/articles/simd-friendly-karp-rabin.html@@ -280,7 +345,7 @@ bool Matcher::simd_advance_string_pma_avx512bw(size_t loc) while (true) { const char *s = buf_ + loc + lcp;- const char *e = buf_ + end_ + lcp - len + 1;+ const char *e = buf_ + end_ + lcp - len - min + 1; __m512i vlcp = _mm512_set1_epi8(chr[lcp]); __m512i vlcs = _mm512_set1_epi8(chr[lcs]); while (s <= e - 64)@@ -293,10 +358,10 @@ bool Matcher::simd_advance_string_pma_avx512bw(size_t loc) uint32_t offset = ctzl(mask); if (std::memcmp(s - lcp + offset, chr, len) == 0) {- loc = s - lcp + offset - buf_;- if (loc + len + 4 > end_ || pat_->predict_match(&buf_[loc + len]))+ size_t k = s - lcp + offset - buf_;+ if (k + len + 4 > end_ || pat_->predict_match(&buf_[k + len])) {- set_current(loc);+ set_current(k); return true; } }@@ -304,16 +369,33 @@ bool Matcher::simd_advance_string_pma_avx512bw(size_t loc) } s += 64; }- s -= lcp;- loc = s - buf_;+ while (s < e)+ {+ do+ s = static_cast<const char*>(std::memchr(s, chr[lcp], e - s));+ while (s != NULL && s[lcs - lcp] != chr[lcs] && ++s < e);+ if (s == NULL || s >= e)+ {+ s = e;+ break;+ }+ if (std::memcmp(s - lcp, chr, len) == 0)+ {+ size_t k = s - lcp - buf_;+ if (k + len + 4 > end_ || pat_->predict_match(&buf_[k + len]))+ {+ set_current(k);+ return true;+ }+ }+ ++s;+ }+ loc = s - lcp - buf_; set_current_and_peek_more(loc); loc = cur_;- if (loc + len + min > end_)+ if (loc + len + min > end_ && eof_) return false;- if (loc + len + min + 63 > end_)- break; }- return advance_string_pma(loc); } /// Implements AVX512BW string search scheme based on http://0x80.pl/articles/simd-friendly-karp-rabin.html@@ -327,7 +409,7 @@ bool Matcher::simd_advance_string_pmh_avx512bw(size_t loc) while (true) { const char *s = buf_ + loc + lcp;- const char *e = buf_ + end_ + lcp - len + 1;+ const char *e = buf_ + end_ + lcp - len - min + 1; __m512i vlcp = _mm512_set1_epi8(chr[lcp]); __m512i vlcs = _mm512_set1_epi8(chr[lcs]); while (s <= e - 64)@@ -340,10 +422,10 @@ bool Matcher::simd_advance_string_pmh_avx512bw(size_t loc) uint32_t offset = ctzl(mask); if (std::memcmp(s - lcp + offset, chr, len) == 0) {- loc = s - lcp + offset - buf_;- if (loc + len + min > end_ || pat_->predict_match(&buf_[loc + len], min))+ size_t k = s - lcp + offset - buf_;+ if (k + len + min > end_ || pat_->predict_match(&buf_[k + len], min)) {- set_current(loc);+ set_current(k); return true; } }@@ -351,16 +433,33 @@ bool Matcher::simd_advance_string_pmh_avx512bw(size_t loc) } s += 64; }- s -= lcp;- loc = s - buf_;+ while (s < e)+ {+ do+ s = static_cast<const char*>(std::memchr(s, chr[lcp], e - s));+ while (s != NULL && s[lcs - lcp] != chr[lcs] && ++s < e);+ if (s == NULL || s >= e)+ {+ s = e;+ break;+ }+ if (std::memcmp(s - lcp, chr, len) == 0)+ {+ size_t k = s - lcp - buf_;+ if (pat_->predict_match(&buf_[k + len], min))+ {+ set_current(k);+ return true;+ }+ }+ ++s;+ }+ loc = s - lcp - buf_; set_current_and_peek_more(loc); loc = cur_;- if (loc + len + min > end_)+ if (loc + len + min > end_ && eof_) return false;- if (loc + len + min + 63 > end_)- break; }- return advance_string_pmh(loc); } #else
@@ -18,10 +18,10 @@ cat >> man/ugrep.1 << 'END' \fBugrep\fR, \fBug\fR -- file pattern searcher .SH SYNOPSIS .B ugrep-[\fIOPTIONS\fR] [\fB-i\fR] [\fB-Q\fR|\fIPATTERN\fR] [\fB-e\fR \fIPATTERN\fR] [\fB-N\fR \fIPATTERN\fR] [\fB-f\fR \fIFILE\fR]- [\fB-F\fR|\fB-G\fR|\fB-P\fR|\fB-Z\fR] [\fB-U\fR] [\fB-m\fR [\fIMIN,\fR][\fIMAX\fR]] [\fB--bool\fR [\fB--files\fR|\fB--lines\fR]]- [\fB-r\fR|\fB-R\fR|\fB-1\fR|...|\fB-9\fR|\fB-10\fR|...] [\fB-t\fR \fITYPES\fR] [\fB-g\fR \fIGLOBS\fR] [\fB--sort\fR[=\fIKEY\fR]]- [\fB-l\fR|\fB-c\fR] [\fB-o\fR] [\fB-n\fR] [\fB-k\fR] [\fB-b\fR] [\fB-A\fR \fINUM\fR] [\fB-B\fR \fINUM\fR] [\fB-C \fR\fINUM\fR] [\fB-y\fR]+[\fIOPTIONS\fR] [\fB-i\fR|\fB-j\fR] [\fB-Q\fR|\fIPATTERN\fR] [\fB-e\fR \fIPATTERN\fR] [\fB-N\fR \fIPATTERN\fR] [\fB-f\fR \fIFILE\fR]+ [\fB-E\fR|\fB-F\fR|\fB-G\fR|\fB-P\fR|\fB-Z\fR] [\fB-U\fR] [\fB-m\fR [\fIMIN,\fR][\fIMAX\fR]] [\fB--bool\fR [\fB--files\fR|\fB--lines\fR]]+ [\fB-r\fR|\fB-R\fR|\fB-1\fR|...|\fB-9\fR|\fB-10\fR|\fB-11\fR|...] [\fB-t\fR \fITYPES\fR] [\fB-g\fR \fIGLOBS\fR] [\fB--sort\fR[=\fIKEY\fR]]+ [\fB-q\fR|\fB-l\fR|\fB-c\fR|\fB-o\fR] [\fB-v\fR] [\fB-n\fR] [\fB-k\fR] [\fB-b\fR] [\fB-A\fR \fINUM\fR] [\fB-B\fR \fINUM\fR] [\fB-C \fR\fINUM\fR] [\fB-y\fR] [\fB--color\fR[=\fIWHEN\fR]|\fB--colour\fR[=\fIWHEN\fR]] [\fB--pretty\fR] [\fB--pager\fR[=\fICOMMAND\fR]] [\fB--hexdump\fR|\fB--csv\fR|\fB--json\fR|\fB--xml\fR] [\fB-I\fR] [\fB-z\fR] [\fB--zmax\fR=\fINUM\fR] [\fIFILE\fR \fI...\fR] .SH DESCRIPTION@@ -56,11 +56,9 @@ search queries with option \fB-%\fR (\fB--bool\fR) to match lines and \fB-%%\fR (\fB--fixed-strings\fR), \fB-G\fR (\fB--basic-regexp\fR) and \fB-P\fR (\fB--perl-regexp\fR) specify other pattern syntaxes. .PP-Option \fB-i\fR (\fB--ignore-case\fR) ignores case in ASCII patterns. When-combined with option \fB-P\fR, ignores case in Unicode patterns. Option+Option \fB-i\fR (\fB--ignore-case\fR) ignores letter case in patterns. Option \fB-j\fR (\fB--smart-case\fR) enables \fB-i\fR only if the search patterns are-specified-in lower case.+specified in lower case. .PP Fuzzy (approximate) search is specified with option \fB-Z\fR (\fB--fuzzy\fR) with an optional argument to control character insertions, deletions, and/or@@ -115,22 +113,23 @@ Compressed files and archives are searched with option \fB-z\fR the contents of compressed files and archives stored within archives up to \fINUM\fR levels. .PP+Output to a terminal for viewing is enhanced with \fB--pretty\fR, which is+enabled by default with the \fBug\fR command.+.PP A query terminal user interface (TUI) is opened with \fB-Q\fR (\fB--query\fR) to interactively specify search patterns and view search results. A \fIPATTERN\fR argument requires \fB-e PATTERN\fR to start the query TUI with the specified pattern. .PP-Output to a terminal for viewing is enhanced with \fB--pretty\fR, which is-enabled by default with the \fBug\fR command.-.PP-A terminal output pager is enabled with \fB--pager\fR.+A terminal output pager is enabled with \fB--pager\fR that accepts an+optional \fICOMMAND\fR to view the output. .PP Customized output is produced with option \fB--format\fR or \fB--replace\fR. Try \fBug --help format\fR for help with custom formatting of the output.-Predefined formats include CSV with option \fB--csv\fR, JSON with option-\fB--json\fR, and XML with option \fB--xml\fR. Hexdumps are output with option-\fB-X\fR (\fB--hex\fR) or with option \fB--hexdump\fR to customize hexdumps.-See also section FORMAT.+See also section FORMAT. Predefined formats include CSV with option+\fB--csv\fR, JSON with option \fB--json\fR, and XML with option \fB--xml\fR.+Hexdumps are output with option \fB-X\fR (\fB--hex\fR) or with option+\fB--hexdump\fR to customize hexdumps. .PP A `--' signals the end of options; the rest of the parameters are \fIFILE\fR arguments, allowing filenames to begin with a `-' character.@@ -206,11 +205,11 @@ when combined with other options. The \fB--save-config\fR=\fIFILE\fR option saves the configuration to \fIFILE\fR. The configuration is written to standard output when \fIFILE\fR is a `-'. .SH GLOBBING-Globbing is used by options \fB-g\fR, \fB--include\fR, \fB--include-dir\fR,-\fB--include-from\fR, \fB--exclude\fR, \fB--exclude-dir\fR,-\fB--exclude-from\fR and \fB--ignore-files\fR to match pathnames and basenames-in recursive searches. Glob arguments for these options should be quoted to-prevent shell globbing.+Globbing is used by options \fB-g\fR (\fB--glob\fR and \fB--iglob\fR),+\fB--include\fR, \fB--include-dir\fR, \fB--include-from\fR, \fB--exclude\fR,+\fB--exclude-dir\fR, \fB--exclude-from\fR and \fB--ignore-files\fR to match+pathnames or basenames in recursive searches. Glob arguments for these+options should be quoted to prevent shell globbing. .PP Globbing supports gitignore syntax and the corresponding matching rules, except that a glob normally matches files but not directories. If a glob ends in a@@ -219,8 +218,8 @@ path separator `/', then it matches directories but not files, as if a path separator `/', the full pathname is matched. Otherwise the basename of a file or directory is matched. For example, \fB*.h\fR matches foo.h and bar/foo.h. \fBbar/*.h\fR matches bar/foo.h but not foo.h and not-bar/bar/foo.h. Use a leading `/' to force \fB/*.h\fR to match foo.h but not-bar/foo.h.+bar/bar/foo.h. Use a leading `./' or just `/' to force \fB/*.h\fR to match+foo.h in the working directory but not bar/foo.h. .PP When a glob starts with a `^' or a `!' as in \fB-g\fR^\fIGLOB\fR, the match is negated. Likewise, a `!' (but not a `^') may be used with globs in the files@@ -240,7 +239,7 @@ Matches one character not a,b,c,d,e,/. .IP \fB[!abc-e]\fR Matches one character not a,b,c,d,e,/. .IP \fB/\fR-When used at the start of a glob, matches if pathname has no /.+When used at the start of a glob, matches the working directory. When used at the end of a glob, matches directories only. .IP \fB**/\fR Matches zero or more directories.@@ -293,6 +292,7 @@ by a previous glob pattern specified in the files associated with .IP \fBGREP_PATH\fR May be used to specify a file path to pattern files. The file path is used by option \fB-f\fR to open a pattern file, when the pattern file does not exist.+Defaults to /usr/local/share/ugrep/patterns when \fBGREP_PATH\fR is undefined. .IP \fBGREP_COLOR\fR May be used to specify ANSI SGR parameters to highlight matches when option \fB--color\fR is used, e.g. 1;35;40 shows pattern matches in bold magenta text@@ -358,7 +358,8 @@ TUI regex lists and literals. .IP \fBqb=\fR TUI regex braces. .SH FORMAT-Option \fB--format\fR=\fIFORMAT\fR specifies an output format for file matches.+Option \fB--format\fR=\fIFORMAT\fR specifies an output format for matches.+Option \fB--replace\fR=\fIFORMAT\fR only replaces the matching patterns. Fields may be used in \fIFORMAT\fR, which expand into the following values: .IP \fB%[\fR\fITEXT\fR\fB]F\fR if option \fB-H\fR is used: \fITEXT\fR, the file pathname and separator.@@ -551,11 +552,11 @@ $ ugrep -cowi patricia myfile.txt .PP List lines with `amount' and a decimal, ignoring case (space is AND): .IP-$ ugrep -i -% 'amount \d+(\.\d+)?' myfile.txt+$ ugrep -i -% 'amount \\d+(\\.\\d+)?' myfile.txt .PP-Alternative query:+The same Boolean search query, but with options \fB-e\fR and \fB--and\fR: .IP-$ ugrep -wi -e amount --and '\d+(\.\d+)?' myfile.txt+$ ugrep -wi -e amount --and '\\d+(\\.\\d+)?' myfile.txt .PP List all Unicode words in a file: .IP@@ -579,7 +580,6 @@ recursive search, showing only matches with up to 10 characters of context before and after: .IP $ ugrep -o -C20 -R -n -k -tjs FIXME- .PP Find blocks of text between lines matching BEGIN and END by using a lazy quantifier `*?' to match only what is necessary and pattern `\\n' to match@@ -668,13 +668,14 @@ recursively below up to two levels deeper (3 levels total): .IP $ ugrep -3 -l -tRpm '' rpm/ .PP-Monitor the system log for bug reports and ungroup multiple matches on a line:+Monitor the system log for bug reports and ungroup multiple matches from+matching lines into separate lines: .IP $ tail -f /var/log/system.log | ugrep -u -i -w bug .PP Interactive fuzzy search with Boolean search queries: .IP-$ ugrep -Q -l -% -Z3 --sort=best+$ ugrep -Q -l -% -Z2 --sort=best .PP Display all words in a MacRoman-encoded file that has CR newlines: .IP@@ -685,7 +686,7 @@ Display options related to "fuzzy" searching: $ ugrep --help fuzzy .PP .SH COPYRIGHT-Copyright (c) 2021,2025 Robert A. van Engelen <engelen@acm.org>+Copyright (c) 2021-2025 Robert A. van Engelen <engelen@acm.org> .PP \fBugrep\fR is released under the BSD\-3 license. All parts of the software have reasonable copyright terms permitting free redistribution. This includes
@@ -1,12 +1,12 @@-.TH UGREP "1" "March 03, 2025" "ugrep 7.3.0" "User Commands"+.TH UGREP "1" "April 21, 2025" "ugrep 7.4.0" "User Commands" .SH NAME \fBugrep\fR, \fBug\fR -- file pattern searcher .SH SYNOPSIS .B ugrep-[\fIOPTIONS\fR] [\fB-i\fR] [\fB-Q\fR|\fIPATTERN\fR] [\fB-e\fR \fIPATTERN\fR] [\fB-N\fR \fIPATTERN\fR] [\fB-f\fR \fIFILE\fR]- [\fB-F\fR|\fB-G\fR|\fB-P\fR|\fB-Z\fR] [\fB-U\fR] [\fB-m\fR [\fIMIN,\fR][\fIMAX\fR]] [\fB--bool\fR [\fB--files\fR|\fB--lines\fR]]- [\fB-r\fR|\fB-R\fR|\fB-1\fR|...|\fB-9\fR|\fB-10\fR|...] [\fB-t\fR \fITYPES\fR] [\fB-g\fR \fIGLOBS\fR] [\fB--sort\fR[=\fIKEY\fR]]- [\fB-l\fR|\fB-c\fR] [\fB-o\fR] [\fB-n\fR] [\fB-k\fR] [\fB-b\fR] [\fB-A\fR \fINUM\fR] [\fB-B\fR \fINUM\fR] [\fB-C \fR\fINUM\fR] [\fB-y\fR]+[\fIOPTIONS\fR] [\fB-i\fR|\fB-j\fR] [\fB-Q\fR|\fIPATTERN\fR] [\fB-e\fR \fIPATTERN\fR] [\fB-N\fR \fIPATTERN\fR] [\fB-f\fR \fIFILE\fR]+ [\fB-E\fR|\fB-F\fR|\fB-G\fR|\fB-P\fR|\fB-Z\fR] [\fB-U\fR] [\fB-m\fR [\fIMIN,\fR][\fIMAX\fR]] [\fB--bool\fR [\fB--files\fR|\fB--lines\fR]]+ [\fB-r\fR|\fB-R\fR|\fB-1\fR|...|\fB-9\fR|\fB-10\fR|\fB-11\fR|...] [\fB-t\fR \fITYPES\fR] [\fB-g\fR \fIGLOBS\fR] [\fB--sort\fR[=\fIKEY\fR]]+ [\fB-q\fR|\fB-l\fR|\fB-c\fR|\fB-o\fR] [\fB-v\fR] [\fB-n\fR] [\fB-k\fR] [\fB-b\fR] [\fB-A\fR \fINUM\fR] [\fB-B\fR \fINUM\fR] [\fB-C \fR\fINUM\fR] [\fB-y\fR] [\fB--color\fR[=\fIWHEN\fR]|\fB--colour\fR[=\fIWHEN\fR]] [\fB--pretty\fR] [\fB--pager\fR[=\fICOMMAND\fR]] [\fB--hexdump\fR|\fB--csv\fR|\fB--json\fR|\fB--xml\fR] [\fB-I\fR] [\fB-z\fR] [\fB--zmax\fR=\fINUM\fR] [\fIFILE\fR \fI...\fR] .SH DESCRIPTION@@ -41,11 +41,9 @@ search queries with option \fB-%\fR (\fB--bool\fR) to match lines and \fB-%%\fR (\fB--fixed-strings\fR), \fB-G\fR (\fB--basic-regexp\fR) and \fB-P\fR (\fB--perl-regexp\fR) specify other pattern syntaxes. .PP-Option \fB-i\fR (\fB--ignore-case\fR) ignores case in ASCII patterns. When-combined with option \fB-P\fR, ignores case in Unicode patterns. Option+Option \fB-i\fR (\fB--ignore-case\fR) ignores letter case in patterns. Option \fB-j\fR (\fB--smart-case\fR) enables \fB-i\fR only if the search patterns are-specified-in lower case.+specified in lower case. .PP Fuzzy (approximate) search is specified with option \fB-Z\fR (\fB--fuzzy\fR) with an optional argument to control character insertions, deletions, and/or@@ -100,22 +98,23 @@ Compressed files and archives are searched with option \fB-z\fR the contents of compressed files and archives stored within archives up to \fINUM\fR levels. .PP+Output to a terminal for viewing is enhanced with \fB--pretty\fR, which is+enabled by default with the \fBug\fR command.+.PP A query terminal user interface (TUI) is opened with \fB-Q\fR (\fB--query\fR) to interactively specify search patterns and view search results. A \fIPATTERN\fR argument requires \fB-e PATTERN\fR to start the query TUI with the specified pattern. .PP-Output to a terminal for viewing is enhanced with \fB--pretty\fR, which is-enabled by default with the \fBug\fR command.-.PP-A terminal output pager is enabled with \fB--pager\fR.+A terminal output pager is enabled with \fB--pager\fR that accepts an+optional \fICOMMAND\fR to view the output. .PP Customized output is produced with option \fB--format\fR or \fB--replace\fR. Try \fBug --help format\fR for help with custom formatting of the output.-Predefined formats include CSV with option \fB--csv\fR, JSON with option-\fB--json\fR, and XML with option \fB--xml\fR. Hexdumps are output with option-\fB-X\fR (\fB--hex\fR) or with option \fB--hexdump\fR to customize hexdumps.-See also section FORMAT.+See also section FORMAT. Predefined formats include CSV with option+\fB--csv\fR, JSON with option \fB--json\fR, and XML with option \fB--xml\fR.+Hexdumps are output with option \fB-X\fR (\fB--hex\fR) or with option+\fB--hexdump\fR to customize hexdumps. .PP A `--' signals the end of options; the rest of the parameters are \fIFILE\fR arguments, allowing filenames to begin with a `-' character.@@ -312,7 +311,7 @@ the default. Specify a PATTERN to search the input. An input line is selected if it matches any of the specified patterns. This option is useful when multiple \fB\-e\fR options are used to specify multiple patterns, or-when a pattern begins with a dash (`\-'), or to specify a pattern+when a pattern starts with a dash (`\-'), or to specify a pattern after option \fB\-f\fR or after the FILE arguments. .TP \fB\-\-encoding\fR=\fIENCODING\fR@@ -342,7 +341,7 @@ quoted to prevent shell globbing. This option may be repeated. .TP \fB\-\-exclude\-dir\fR=\fIGLOB\fR Exclude directories whose name matches GLOB from recursive-searches, same as \fB\-g\fR ^GLOB/. GLOB can use **, *, ?, and [...] as+searches, same as \fB\-g\fR ^GLOB/. GLOB may use **, *, ?, and [...] as wildcards and \\ to quote a wildcard or backslash character literally. When GLOB contains a `/', full pathnames are matched. Otherwise basenames are matched. Note that \fB\-\-exclude\-dir\fR patterns@@ -351,7 +350,7 @@ to prevent shell globbing. This option may be repeated. .TP \fB\-\-exclude\-from\fR=\fIFILE\fR Read the globs from FILE and skip files and directories whose name-matches one or more globs. A glob can use **, *, ?, and [...] as+matches one or more globs. A glob may use **, *, ?, and [...] as wildcards and \\ to quote a wildcard or backslash character literally. When a glob contains a `/', full pathnames are matched. Otherwise basenames are matched. When a glob ends with a `/',@@ -419,21 +418,28 @@ is also enabled. Context options \fB\-A\fR, \fB\-B\fR, \fB\-C\fR and \fB\-y\fR \fB\-\-free\-space\fR Spacing (blanks and tabs) in regular expressions are ignored. .TP+\fB\-\-from\fR=\fIFILE\fR+Read additional pathnames of files to search from FILE. When FILE+is a `\-', standard input is read. This option is useful with `find+... \fB\-p\fRrint | ugrep \fB\-\-from\fR=\fI\-\fR ...' to search specific files.+.TP \fB\-G\fR, \fB\-\-basic\-regexp\fR Interpret patterns as basic regular expressions (BREs). .TP \fB\-g\fR \fIGLOBS\fR, \fB\-\-glob\fR=\fIGLOBS\fR, \fB\-\-iglob\fR=\fIGLOBS\fR Only search files whose name matches the specified comma\-separated-list of GLOBS, same as \fB\-\-include\fR=glob for each `glob' in GLOBS.+list of GLOBS, same as \fB\-\-include\fR='glob' for each `glob' in GLOBS. When a `glob' is preceded by a `!' or a `^', skip files whose name matches `glob', same as \fB\-\-exclude\fR='glob'. When `glob' contains a-`/', full pathnames are matched. Otherwise basenames are matched.-When `glob' ends with a `/', directories are matched, same as-\fB\-\-include\-dir\fR='glob' and \fB\-\-exclude\-dir\fR='glob'. A leading `/'-matches the working directory. Option \fB\-\-iglob\fR performs-case\-insensitive name matching. This option may be repeated and-may be combined with options \fB\-M\fR, \fB\-O\fR and \fB\-t\fR. For more details, see-`ugrep \fB\-\-help\fR globs' and `man ugrep' section GLOBBING for details.+`/', full pathnames are matched and option \fB\-r\fR is enabled to recurse+into directories if \fB\-d\fR, \fB\-r\fR or \fB\-R\fR is not already specified.+Otherwise basenames are matched. When `glob' ends with a `/',+directories are matched, same as \fB\-\-include\-dir\fR='glob' and+\fB\-\-exclude\-dir\fR='glob'. A leading `./' or just `/' matches the+working directory. Option \fB\-\-iglob\fR=\fIGLOBS\fR performs case\-insensitive+name matching. This option may be repeated and may be combined+with options \fB\-M\fR, \fB\-O\fR and \fB\-t\fR. For more details, see `ugrep \fB\-\-help\fR+globs' and `man ugrep' section GLOBBING for details. .TP \fB\-\-glob\-ignore\-case\fR Perform case\-insensitive glob matching in general.@@ -512,26 +518,30 @@ Do not ignore files, i.e. cancel \fB\-\-ignore\-files\fR when specified. .TP \fB\-\-include\fR=\fIGLOB\fR Only search files whose name matches GLOB, same as \fB\-g\fR \fIGLOB\fR. GLOB-can use **, *, ?, and [...] as wildcards and \\ to quote a wildcard+may use **, *, ?, and [...] as wildcards and \\ to quote a wildcard or backslash character literally. When GLOB contains a `/', full-pathnames are matched. Otherwise basenames are matched. When GLOB-ends with a `/', directories are included as if \fB\-\-include\-dir\fR is-specified. Otherwise files are included. Note that \fB\-\-exclude\fR-patterns take priority over \fB\-\-include\fR patterns. GLOB should be-quoted to prevent shell globbing. This option may be repeated.+pathnames are matched and option \fB\-r\fR is enabled to recurse into+directories if \fB\-d\fR, \fB\-r\fR or \fB\-R\fR is not already specified. Otherwise+basenames are matched. When GLOB ends with a `/', directories are+included as if \fB\-\-include\-dir\fR is specified. Otherwise files are+included. Note that \fB\-\-exclude\fR patterns take priority over+\fB\-\-include\fR patterns. GLOB should be quoted to prevent shell+globbing. This option may be repeated. .TP \fB\-\-include\-dir\fR=\fIGLOB\fR Only directories whose name matches GLOB are included in recursive-searches, same as \fB\-g\fR \fIGLOB\fR/. GLOB can use **, *, ?, and [...] as+searches, same as \fB\-g\fR \fIGLOB\fR/. GLOB may use **, *, ?, and [...] as wildcards and \\ to quote a wildcard or backslash character-literally. When GLOB contains a `/', full pathnames are matched.-Otherwise basenames are matched. Note that \fB\-\-exclude\-dir\fR patterns-take priority over \fB\-\-include\-dir\fR patterns. GLOB should be quoted-to prevent shell globbing. This option may be repeated.+literally. When GLOB contains a `/', full pathnames are matched+and option \fB\-r\fR is enabled to recurse into directories if \fB\-d\fR, \fB\-r\fR or+\fB\-R\fR is not already specified. Otherwise basenames are matched.+Note that \fB\-\-exclude\-dir\fR patterns take priority over \fB\-\-include\-dir\fR+patterns. GLOB should be quoted to prevent shell globbing. This+option may be repeated. .TP \fB\-\-include\-from\fR=\fIFILE\fR Read the globs from FILE and search only files and directories-whose name matches one or more globs. A glob can use **, *, ?, and+whose name matches one or more globs. A glob may use **, *, ?, and [...] as wildcards and \\ to quote a wildcard or backslash character literally. When a glob contains a `/', full pathnames are matched. Otherwise basenames are matched. When a glob ends@@ -729,7 +739,7 @@ the default is 3 (300ms delay). No whitespace may be given between \fB\-e\fR \fIPATTERN\fR, i.e. a PATTERN argument requires option \fB\-e\fR. Press F1 or CTRL\-Z to view the help screen. Press F2 or CTRL\-Y to invoke a command to view or edit the file shown at the top of the screen.-The command can be specified with option \fB\-\-view\fR and defaults to+This command can be specified with option \fB\-\-view\fR and defaults to environment variable PAGER when defined, or VISUAL or EDITOR. Press TAB or SHIFT\-TAB to navigate directories and to select a file to search. Press ENTER to select lines to output. Press ALT\-l for@@ -831,10 +841,10 @@ repeated. The possible file types can be (\fB\-t\fRlist displays a list): `parrot', `pascal', `pdf', `Pdf', `perl', `Perl', `php', `Php', `png', `Png', `prolog', `python', `Python', `r', `rpm', `Rpm', `rst', `rtf', `Rtf', `ruby', `Ruby',-`rust', `scala', `scheme', `shell', `Shell', `smalltalk', `sql',-`svg', `swift', `tcl', `tex', `text', `tiff', `Tiff',-`tt', `typescript', `verilog', `vhdl', `vim', `xml', `Xml',-`yacc', `yaml', `zig'.+`rust', `scala', `scheme', `seed7', `shell', `Shell', `smalltalk',+`sql', `svg', `swift', `tcl', `tex', `text', `tiff',+`Tiff', `tt', `typescript', `verilog', `vhdl', `vim', `xml',+`Xml', `yacc', `yaml', `zig'. .TP \fB\-\-tabs\fR[=\fINUM\fR] Set the tab size to NUM to expand tabs for option \fB\-k\fR. The value of@@ -902,10 +912,11 @@ additional values are output. See also options \fB\-\-format\fR and \fB\-u\fR. .TP \fB\-Y\fR, \fB\-\-empty\fR Empty\-matching patterns match all lines. Normally, empty matches-are not output, unless a pattern begins with `^' or ends with `$'.-With this option, empty\-matching patterns, such as x? and x*, match-all lines, not only lines with an `x' (enabled by default in grep-compatibility mode).+are not output, unless a pattern starts with `^' or ends with `$'.+With this option, empty\-matching patterns, such as x*y?, x*y*z*,+and x*(y?|z) match all lines, not only the expected pattern.+This option is enabled by default in grep compatibility mode;+specify option \fB\-\-no\-empty\fR to prevent matching all lines. .TP \fB\-y\fR, \fB\-\-any\-line\fR, \fB\-\-passthru\fR Any line is output (passthru). Non\-matching lines are output as@@ -935,15 +946,15 @@ pathnames of files in archives are output in braces. When used with option \fB\-\-zmax\fR=\fINUM\fR, searches the contents of compressed files and archives stored within archives up to NUM levels. When \fB\-g\fR, \fB\-O\fR, \fB\-M\fR, or \fB\-t\fR is specified, searches archives for files that match the-specified globs, file extensions, file signature magic bytes, or-file types, respectively; a side\-effect of these options is that+specified globs, filename extensions, file signature magic bytes,+or file types, respectively; a side\-effect of these options is that the compressed files and archives searched are only those with filename extensions that match known compression and archive types. Supported compression formats: gzip (.gz), compress (.Z), zip, 7z,-bzip2 (requires suffix .bz, .bz2, .bzip2, .tbz, .tbz2, .tb2, .tz2),-lzma and xz (requires suffix .lzma, .tlz, .xz, .txz),+bzip2 (.bz, .bz2, .bzip2, .tbz, .tbz2, .tb2, .tz2),+xz (.xz, .txz) and lzma (requires suffix .lzma, .tlz),+zstd (.zst, .zstd, .tzst), lz4 (requires suffix .lz4),-zstd (requires suffix .zst, .zstd, .tzst), brotli (requires suffix .br), bzip3 (requires suffix .bz3). .TP@@ -995,11 +1006,11 @@ when combined with other options. The \fB--save-config\fR=\fIFILE\fR option saves the configuration to \fIFILE\fR. The configuration is written to standard output when \fIFILE\fR is a `-'. .SH GLOBBING-Globbing is used by options \fB-g\fR, \fB--include\fR, \fB--include-dir\fR,-\fB--include-from\fR, \fB--exclude\fR, \fB--exclude-dir\fR,-\fB--exclude-from\fR and \fB--ignore-files\fR to match pathnames and basenames-in recursive searches. Glob arguments for these options should be quoted to-prevent shell globbing.+Globbing is used by options \fB-g\fR (\fB--glob\fR and \fB--iglob\fR),+\fB--include\fR, \fB--include-dir\fR, \fB--include-from\fR, \fB--exclude\fR,+\fB--exclude-dir\fR, \fB--exclude-from\fR and \fB--ignore-files\fR to match+pathnames or basenames in recursive searches. Glob arguments for these+options should be quoted to prevent shell globbing. .PP Globbing supports gitignore syntax and the corresponding matching rules, except that a glob normally matches files but not directories. If a glob ends in a@@ -1008,8 +1019,8 @@ path separator `/', then it matches directories but not files, as if a path separator `/', the full pathname is matched. Otherwise the basename of a file or directory is matched. For example, \fB*.h\fR matches foo.h and bar/foo.h. \fBbar/*.h\fR matches bar/foo.h but not foo.h and not-bar/bar/foo.h. Use a leading `/' to force \fB/*.h\fR to match foo.h but not-bar/foo.h.+bar/bar/foo.h. Use a leading `./' or just `/' to force \fB/*.h\fR to match+foo.h in the working directory but not bar/foo.h. .PP When a glob starts with a `^' or a `!' as in \fB-g\fR^\fIGLOB\fR, the match is negated. Likewise, a `!' (but not a `^') may be used with globs in the files@@ -1029,7 +1040,7 @@ Matches one character not a,b,c,d,e,/. .IP \fB[!abc-e]\fR Matches one character not a,b,c,d,e,/. .IP \fB/\fR-When used at the start of a glob, matches if pathname has no /.+When used at the start of a glob, matches the working directory. When used at the end of a glob, matches directories only. .IP \fB**/\fR Matches zero or more directories.@@ -1082,6 +1093,7 @@ by a previous glob pattern specified in the files associated with .IP \fBGREP_PATH\fR May be used to specify a file path to pattern files. The file path is used by option \fB-f\fR to open a pattern file, when the pattern file does not exist.+Defaults to /usr/local/share/ugrep/patterns when \fBGREP_PATH\fR is undefined. .IP \fBGREP_COLOR\fR May be used to specify ANSI SGR parameters to highlight matches when option \fB--color\fR is used, e.g. 1;35;40 shows pattern matches in bold magenta text@@ -1147,7 +1159,8 @@ TUI regex lists and literals. .IP \fBqb=\fR TUI regex braces. .SH FORMAT-Option \fB--format\fR=\fIFORMAT\fR specifies an output format for file matches.+Option \fB--format\fR=\fIFORMAT\fR specifies an output format for matches.+Option \fB--replace\fR=\fIFORMAT\fR only replaces the matching patterns. Fields may be used in \fIFORMAT\fR, which expand into the following values: .IP \fB%[\fR\fITEXT\fR\fB]F\fR if option \fB-H\fR is used: \fITEXT\fR, the file pathname and separator.@@ -1340,11 +1353,11 @@ $ ugrep -cowi patricia myfile.txt .PP List lines with `amount' and a decimal, ignoring case (space is AND): .IP-$ ugrep -i -% 'amount \d+(\.\d+)?' myfile.txt+$ ugrep -i -% 'amount \\d+(\\.\\d+)?' myfile.txt .PP-Alternative query:+The same Boolean search query, but with options \fB-e\fR and \fB--and\fR: .IP-$ ugrep -wi -e amount --and '\d+(\.\d+)?' myfile.txt+$ ugrep -wi -e amount --and '\\d+(\\.\\d+)?' myfile.txt .PP List all Unicode words in a file: .IP@@ -1368,7 +1381,6 @@ recursive search, showing only matches with up to 10 characters of context before and after: .IP $ ugrep -o -C20 -R -n -k -tjs FIXME- .PP Find blocks of text between lines matching BEGIN and END by using a lazy quantifier `*?' to match only what is necessary and pattern `\\n' to match@@ -1457,13 +1469,14 @@ recursively below up to two levels deeper (3 levels total): .IP $ ugrep -3 -l -tRpm '' rpm/ .PP-Monitor the system log for bug reports and ungroup multiple matches on a line:+Monitor the system log for bug reports and ungroup multiple matches from+matching lines into separate lines: .IP $ tail -f /var/log/system.log | ugrep -u -i -w bug .PP Interactive fuzzy search with Boolean search queries: .IP-$ ugrep -Q -l -% -Z3 --sort=best+$ ugrep -Q -l -% -Z2 --sort=best .PP Display all words in a MacRoman-encoded file that has CR newlines: .IP@@ -1474,7 +1487,7 @@ Display options related to "fuzzy" searching: $ ugrep --help fuzzy .PP .SH COPYRIGHT-Copyright (c) 2021,2025 Robert A. van Engelen <engelen@acm.org>+Copyright (c) 2021-2025 Robert A. van Engelen <engelen@acm.org> .PP \fBugrep\fR is released under the BSD\-3 license. All parts of the software have reasonable copyright terms permitting free redistribution. This includes
@@ -1,4 +1,4 @@-.TH UGREP-INDEXER "1" "March 03, 2025" "ugrep-indexer 7.3.0" "User Commands"+.TH UGREP-INDEXER "1" "April 21, 2025" "ugrep-indexer 7.4.0" "User Commands" .SH NAME \fBugrep-indexer\fR -- file indexer to accelerate recursive searching .SH SYNOPSIS
@@ -1,12 +1,12 @@-.TH UGREP "1" "March 03, 2025" "ugrep 7.3.0" "User Commands"+.TH UGREP "1" "April 21, 2025" "ugrep 7.4.0" "User Commands" .SH NAME \fBugrep\fR, \fBug\fR -- file pattern searcher .SH SYNOPSIS .B ugrep-[\fIOPTIONS\fR] [\fB-i\fR] [\fB-Q\fR|\fIPATTERN\fR] [\fB-e\fR \fIPATTERN\fR] [\fB-N\fR \fIPATTERN\fR] [\fB-f\fR \fIFILE\fR]- [\fB-F\fR|\fB-G\fR|\fB-P\fR|\fB-Z\fR] [\fB-U\fR] [\fB-m\fR [\fIMIN,\fR][\fIMAX\fR]] [\fB--bool\fR [\fB--files\fR|\fB--lines\fR]]- [\fB-r\fR|\fB-R\fR|\fB-1\fR|...|\fB-9\fR|\fB-10\fR|...] [\fB-t\fR \fITYPES\fR] [\fB-g\fR \fIGLOBS\fR] [\fB--sort\fR[=\fIKEY\fR]]- [\fB-l\fR|\fB-c\fR] [\fB-o\fR] [\fB-n\fR] [\fB-k\fR] [\fB-b\fR] [\fB-A\fR \fINUM\fR] [\fB-B\fR \fINUM\fR] [\fB-C \fR\fINUM\fR] [\fB-y\fR]+[\fIOPTIONS\fR] [\fB-i\fR|\fB-j\fR] [\fB-Q\fR|\fIPATTERN\fR] [\fB-e\fR \fIPATTERN\fR] [\fB-N\fR \fIPATTERN\fR] [\fB-f\fR \fIFILE\fR]+ [\fB-E\fR|\fB-F\fR|\fB-G\fR|\fB-P\fR|\fB-Z\fR] [\fB-U\fR] [\fB-m\fR [\fIMIN,\fR][\fIMAX\fR]] [\fB--bool\fR [\fB--files\fR|\fB--lines\fR]]+ [\fB-r\fR|\fB-R\fR|\fB-1\fR|...|\fB-9\fR|\fB-10\fR|\fB-11\fR|...] [\fB-t\fR \fITYPES\fR] [\fB-g\fR \fIGLOBS\fR] [\fB--sort\fR[=\fIKEY\fR]]+ [\fB-q\fR|\fB-l\fR|\fB-c\fR|\fB-o\fR] [\fB-v\fR] [\fB-n\fR] [\fB-k\fR] [\fB-b\fR] [\fB-A\fR \fINUM\fR] [\fB-B\fR \fINUM\fR] [\fB-C \fR\fINUM\fR] [\fB-y\fR] [\fB--color\fR[=\fIWHEN\fR]|\fB--colour\fR[=\fIWHEN\fR]] [\fB--pretty\fR] [\fB--pager\fR[=\fICOMMAND\fR]] [\fB--hexdump\fR|\fB--csv\fR|\fB--json\fR|\fB--xml\fR] [\fB-I\fR] [\fB-z\fR] [\fB--zmax\fR=\fINUM\fR] [\fIFILE\fR \fI...\fR] .SH DESCRIPTION@@ -41,11 +41,9 @@ search queries with option \fB-%\fR (\fB--bool\fR) to match lines and \fB-%%\fR (\fB--fixed-strings\fR), \fB-G\fR (\fB--basic-regexp\fR) and \fB-P\fR (\fB--perl-regexp\fR) specify other pattern syntaxes. .PP-Option \fB-i\fR (\fB--ignore-case\fR) ignores case in ASCII patterns. When-combined with option \fB-P\fR, ignores case in Unicode patterns. Option+Option \fB-i\fR (\fB--ignore-case\fR) ignores letter case in patterns. Option \fB-j\fR (\fB--smart-case\fR) enables \fB-i\fR only if the search patterns are-specified-in lower case.+specified in lower case. .PP Fuzzy (approximate) search is specified with option \fB-Z\fR (\fB--fuzzy\fR) with an optional argument to control character insertions, deletions, and/or@@ -57,7 +55,7 @@ one or more newline characters. .PP The empty pattern "" matches all lines. Other empty-matching patterns do not. For example, the pattern `a*' will match one or more a's. Option \fB-Y\fR-forces empty matches for compatibility with other grep tools.+forces empty matches for compatibility with other grep tools. .PP Option \fB-f\fR \fIFILE\fR matches patterns specified in \fIFILE\fR. .PP@@ -100,22 +98,23 @@ Compressed files and archives are searched with option \fB-z\fR the contents of compressed files and archives stored within archives up to \fINUM\fR levels. .PP+Output to a terminal for viewing is enhanced with \fB--pretty\fR, which is+enabled by default with the \fBug\fR command.+.PP A query terminal user interface (TUI) is opened with \fB-Q\fR (\fB--query\fR) to interactively specify search patterns and view search results. A \fIPATTERN\fR argument requires \fB-e PATTERN\fR to start the query TUI with the specified pattern. .PP-Output to a terminal for viewing is enhanced with \fB--pretty\fR, which is-enabled by default with the \fBug\fR command.-.PP-A terminal output pager is enabled with \fB--pager\fR.+A terminal output pager is enabled with \fB--pager\fR that accepts an+optional \fICOMMAND\fR to view the output. .PP Customized output is produced with option \fB--format\fR or \fB--replace\fR. Try \fBug --help format\fR for help with custom formatting of the output.-Predefined formats include CSV with option \fB--csv\fR, JSON with option-\fB--json\fR, and XML with option \fB--xml\fR. Hexdumps are output with option-\fB-X\fR (\fB--hex\fR) or with option \fB--hexdump\fR to customize hexdumps.-See also section FORMAT.+See also section FORMAT. Predefined formats include CSV with option+\fB--csv\fR, JSON with option \fB--json\fR, and XML with option \fB--xml\fR.+Hexdumps are output with option \fB-X\fR (\fB--hex\fR) or with option+\fB--hexdump\fR to customize hexdumps. .PP A `--' signals the end of options; the rest of the parameters are \fIFILE\fR arguments, allowing filenames to begin with a `-' character.@@ -312,7 +311,7 @@ the default. Specify a PATTERN to search the input. An input line is selected if it matches any of the specified patterns. This option is useful when multiple \fB\-e\fR options are used to specify multiple patterns, or-when a pattern begins with a dash (`\-'), or to specify a pattern+when a pattern starts with a dash (`\-'), or to specify a pattern after option \fB\-f\fR or after the FILE arguments. .TP \fB\-\-encoding\fR=\fIENCODING\fR@@ -342,7 +341,7 @@ quoted to prevent shell globbing. This option may be repeated. .TP \fB\-\-exclude\-dir\fR=\fIGLOB\fR Exclude directories whose name matches GLOB from recursive-searches, same as \fB\-g\fR ^GLOB/. GLOB can use **, *, ?, and [...] as+searches, same as \fB\-g\fR ^GLOB/. GLOB may use **, *, ?, and [...] as wildcards and \\ to quote a wildcard or backslash character literally. When GLOB contains a `/', full pathnames are matched. Otherwise basenames are matched. Note that \fB\-\-exclude\-dir\fR patterns@@ -351,7 +350,7 @@ to prevent shell globbing. This option may be repeated. .TP \fB\-\-exclude\-from\fR=\fIFILE\fR Read the globs from FILE and skip files and directories whose name-matches one or more globs. A glob can use **, *, ?, and [...] as+matches one or more globs. A glob may use **, *, ?, and [...] as wildcards and \\ to quote a wildcard or backslash character literally. When a glob contains a `/', full pathnames are matched. Otherwise basenames are matched. When a glob ends with a `/',@@ -419,21 +418,28 @@ is also enabled. Context options \fB\-A\fR, \fB\-B\fR, \fB\-C\fR and \fB\-y\fR \fB\-\-free\-space\fR Spacing (blanks and tabs) in regular expressions are ignored. .TP+\fB\-\-from\fR=\fIFILE\fR+Read additional pathnames of files to search from FILE. When FILE+is a `\-', standard input is read. This option is useful with `find+... \fB\-p\fRrint | ugrep \fB\-\-from\fR=\fI\-\fR ...' to search specific files.+.TP \fB\-G\fR, \fB\-\-basic\-regexp\fR Interpret patterns as basic regular expressions (BREs). .TP \fB\-g\fR \fIGLOBS\fR, \fB\-\-glob\fR=\fIGLOBS\fR, \fB\-\-iglob\fR=\fIGLOBS\fR Only search files whose name matches the specified comma\-separated-list of GLOBS, same as \fB\-\-include\fR=glob for each `glob' in GLOBS.+list of GLOBS, same as \fB\-\-include\fR='glob' for each `glob' in GLOBS. When a `glob' is preceded by a `!' or a `^', skip files whose name matches `glob', same as \fB\-\-exclude\fR='glob'. When `glob' contains a-`/', full pathnames are matched. Otherwise basenames are matched.-When `glob' ends with a `/', directories are matched, same as-\fB\-\-include\-dir\fR='glob' and \fB\-\-exclude\-dir\fR='glob'. A leading `/'-matches the working directory. Option \fB\-\-iglob\fR performs-case\-insensitive name matching. This option may be repeated and-may be combined with options \fB\-M\fR, \fB\-O\fR and \fB\-t\fR. For more details, see-`ugrep \fB\-\-help\fR globs' and `man ugrep' section GLOBBING for details.+`/', full pathnames are matched and option \fB\-r\fR is enabled to recurse+into directories if \fB\-d\fR, \fB\-r\fR or \fB\-R\fR is not already specified.+Otherwise basenames are matched. When `glob' ends with a `/',+directories are matched, same as \fB\-\-include\-dir\fR='glob' and+\fB\-\-exclude\-dir\fR='glob'. A leading `./' or just `/' matches the+working directory. Option \fB\-\-iglob\fR=\fIGLOBS\fR performs case\-insensitive+name matching. This option may be repeated and may be combined+with options \fB\-M\fR, \fB\-O\fR and \fB\-t\fR. For more details, see `ugrep \fB\-\-help\fR+globs' and `man ugrep' section GLOBBING for details. .TP \fB\-\-glob\-ignore\-case\fR Perform case\-insensitive glob matching in general.@@ -512,26 +518,30 @@ Do not ignore files, i.e. cancel \fB\-\-ignore\-files\fR when specified. .TP \fB\-\-include\fR=\fIGLOB\fR Only search files whose name matches GLOB, same as \fB\-g\fR \fIGLOB\fR. GLOB-can use **, *, ?, and [...] as wildcards and \\ to quote a wildcard+may use **, *, ?, and [...] as wildcards and \\ to quote a wildcard or backslash character literally. When GLOB contains a `/', full-pathnames are matched. Otherwise basenames are matched. When GLOB-ends with a `/', directories are included as if \fB\-\-include\-dir\fR is-specified. Otherwise files are included. Note that \fB\-\-exclude\fR-patterns take priority over \fB\-\-include\fR patterns. GLOB should be-quoted to prevent shell globbing. This option may be repeated.+pathnames are matched and option \fB\-r\fR is enabled to recurse into+directories if \fB\-d\fR, \fB\-r\fR or \fB\-R\fR is not already specified. Otherwise+basenames are matched. When GLOB ends with a `/', directories are+included as if \fB\-\-include\-dir\fR is specified. Otherwise files are+included. Note that \fB\-\-exclude\fR patterns take priority over+\fB\-\-include\fR patterns. GLOB should be quoted to prevent shell+globbing. This option may be repeated. .TP \fB\-\-include\-dir\fR=\fIGLOB\fR Only directories whose name matches GLOB are included in recursive-searches, same as \fB\-g\fR \fIGLOB\fR/. GLOB can use **, *, ?, and [...] as+searches, same as \fB\-g\fR \fIGLOB\fR/. GLOB may use **, *, ?, and [...] as wildcards and \\ to quote a wildcard or backslash character-literally. When GLOB contains a `/', full pathnames are matched.-Otherwise basenames are matched. Note that \fB\-\-exclude\-dir\fR patterns-take priority over \fB\-\-include\-dir\fR patterns. GLOB should be quoted-to prevent shell globbing. This option may be repeated.+literally. When GLOB contains a `/', full pathnames are matched+and option \fB\-r\fR is enabled to recurse into directories if \fB\-d\fR, \fB\-r\fR or+\fB\-R\fR is not already specified. Otherwise basenames are matched.+Note that \fB\-\-exclude\-dir\fR patterns take priority over \fB\-\-include\-dir\fR+patterns. GLOB should be quoted to prevent shell globbing. This+option may be repeated. .TP \fB\-\-include\-from\fR=\fIFILE\fR Read the globs from FILE and search only files and directories-whose name matches one or more globs. A glob can use **, *, ?, and+whose name matches one or more globs. A glob may use **, *, ?, and [...] as wildcards and \\ to quote a wildcard or backslash character literally. When a glob contains a `/', full pathnames are matched. Otherwise basenames are matched. When a glob ends@@ -729,7 +739,7 @@ the default is 3 (300ms delay). No whitespace may be given between \fB\-e\fR \fIPATTERN\fR, i.e. a PATTERN argument requires option \fB\-e\fR. Press F1 or CTRL\-Z to view the help screen. Press F2 or CTRL\-Y to invoke a command to view or edit the file shown at the top of the screen.-The command can be specified with option \fB\-\-view\fR and defaults to+This command can be specified with option \fB\-\-view\fR and defaults to environment variable PAGER when defined, or VISUAL or EDITOR. Press TAB or SHIFT\-TAB to navigate directories and to select a file to search. Press ENTER to select lines to output. Press ALT\-l for@@ -832,9 +842,9 @@ repeated. The possible file types can be (\fB\-t\fRlist displays a list): `Php', `png', `Png', `prolog', `python', `Python', `r', `rpm', `Rpm', `rst', `rtf', `Rtf', `ruby', `Ruby', `rust', `scala', `scheme', `seed7', `shell', `Shell', `smalltalk',-`sql', svg', `swift', `tcl', `tex', `text', `tiff', `Tiff',-`tt', `typescript', `verilog', `vhdl', `vim', `xml', `Xml',-`yacc', `yaml', `zig'.+`sql', `svg', `swift', `tcl', `tex', `text', `tiff',+`Tiff', `tt', `typescript', `verilog', `vhdl', `vim', `xml',+`Xml', `yacc', `yaml', `zig'. .TP \fB\-\-tabs\fR[=\fINUM\fR] Set the tab size to NUM to expand tabs for option \fB\-k\fR. The value of@@ -902,10 +912,11 @@ additional values are output. See also options \fB\-\-format\fR and \fB\-u\fR. .TP \fB\-Y\fR, \fB\-\-empty\fR Empty\-matching patterns match all lines. Normally, empty matches-are not output, unless a pattern begins with `^' or ends with `$'.-With this option, empty\-matching patterns, such as x? and x*, match-all lines, not only lines with an `x' (enabled by default in grep-compatibility mode).+are not output, unless a pattern starts with `^' or ends with `$'.+With this option, empty\-matching patterns, such as x*y?, x*y*z*,+and x*(y?|z) match all lines, not only the expected pattern.+This option is enabled by default in grep compatibility mode;+specify option \fB\-\-no\-empty\fR to prevent matching all lines. .TP \fB\-y\fR, \fB\-\-any\-line\fR, \fB\-\-passthru\fR Any line is output (passthru). Non\-matching lines are output as@@ -935,15 +946,15 @@ pathnames of files in archives are output in braces. When used with option \fB\-\-zmax\fR=\fINUM\fR, searches the contents of compressed files and archives stored within archives up to NUM levels. When \fB\-g\fR, \fB\-O\fR, \fB\-M\fR, or \fB\-t\fR is specified, searches archives for files that match the-specified globs, file extensions, file signature magic bytes, or-file types, respectively; a side\-effect of these options is that+specified globs, filename extensions, file signature magic bytes,+or file types, respectively; a side\-effect of these options is that the compressed files and archives searched are only those with filename extensions that match known compression and archive types. Supported compression formats: gzip (.gz), compress (.Z), zip, 7z,-bzip2 (requires suffix .bz, .bz2, .bzip2, .tbz, .tbz2, .tb2, .tz2),-lzma and xz (requires suffix .lzma, .tlz, .xz, .txz),+bzip2 (.bz, .bz2, .bzip2, .tbz, .tbz2, .tb2, .tz2),+xz (.xz, .txz) and lzma (requires suffix .lzma, .tlz),+zstd (.zst, .zstd, .tzst), lz4 (requires suffix .lz4),-zstd (requires suffix .zst, .zstd, .tzst), brotli (requires suffix .br), bzip3 (requires suffix .bz3). .TP@@ -995,11 +1006,11 @@ when combined with other options. The \fB--save-config\fR=\fIFILE\fR option saves the configuration to \fIFILE\fR. The configuration is written to standard output when \fIFILE\fR is a `-'. .SH GLOBBING-Globbing is used by options \fB-g\fR, \fB--include\fR, \fB--include-dir\fR,-\fB--include-from\fR, \fB--exclude\fR, \fB--exclude-dir\fR,-\fB--exclude-from\fR and \fB--ignore-files\fR to match pathnames and basenames-in recursive searches. Glob arguments for these options should be quoted to-prevent shell globbing.+Globbing is used by options \fB-g\fR (\fB--glob\fR and \fB--iglob\fR),+\fB--include\fR, \fB--include-dir\fR, \fB--include-from\fR, \fB--exclude\fR,+\fB--exclude-dir\fR, \fB--exclude-from\fR and \fB--ignore-files\fR to match+pathnames or basenames in recursive searches. Glob arguments for these+options should be quoted to prevent shell globbing. .PP Globbing supports gitignore syntax and the corresponding matching rules, except that a glob normally matches files but not directories. If a glob ends in a@@ -1008,8 +1019,8 @@ path separator `/', then it matches directories but not files, as if a path separator `/', the full pathname is matched. Otherwise the basename of a file or directory is matched. For example, \fB*.h\fR matches foo.h and bar/foo.h. \fBbar/*.h\fR matches bar/foo.h but not foo.h and not-bar/bar/foo.h. Use a leading `/' to force \fB/*.h\fR to match foo.h but not-bar/foo.h.+bar/bar/foo.h. Use a leading `./' or just `/' to force \fB/*.h\fR to match+foo.h in the working directory but not bar/foo.h. .PP When a glob starts with a `^' or a `!' as in \fB-g\fR^\fIGLOB\fR, the match is negated. Likewise, a `!' (but not a `^') may be used with globs in the files@@ -1029,7 +1040,7 @@ Matches one character not a,b,c,d,e,/. .IP \fB[!abc-e]\fR Matches one character not a,b,c,d,e,/. .IP \fB/\fR-When used at the start of a glob, matches if pathname has no /.+When used at the start of a glob, matches the working directory. When used at the end of a glob, matches directories only. .IP \fB**/\fR Matches zero or more directories.@@ -1082,6 +1093,7 @@ by a previous glob pattern specified in the files associated with .IP \fBGREP_PATH\fR May be used to specify a file path to pattern files. The file path is used by option \fB-f\fR to open a pattern file, when the pattern file does not exist.+Defaults to /usr/local/share/ugrep/patterns when \fBGREP_PATH\fR is undefined. .IP \fBGREP_COLOR\fR May be used to specify ANSI SGR parameters to highlight matches when option \fB--color\fR is used, e.g. 1;35;40 shows pattern matches in bold magenta text@@ -1147,7 +1159,8 @@ TUI regex lists and literals. .IP \fBqb=\fR TUI regex braces. .SH FORMAT-Option \fB--format\fR=\fIFORMAT\fR specifies an output format for file matches.+Option \fB--format\fR=\fIFORMAT\fR specifies an output format for matches.+Option \fB--replace\fR=\fIFORMAT\fR only replaces the matching patterns. Fields may be used in \fIFORMAT\fR, which expand into the following values: .IP \fB%[\fR\fITEXT\fR\fB]F\fR if option \fB-H\fR is used: \fITEXT\fR, the file pathname and separator.@@ -1340,11 +1353,11 @@ $ ugrep -cowi patricia myfile.txt .PP List lines with `amount' and a decimal, ignoring case (space is AND): .IP-$ ugrep -i -% 'amount \d+(\.\d+)?' myfile.txt+$ ugrep -i -% 'amount \\d+(\\.\\d+)?' myfile.txt .PP-Alternative query:+The same Boolean search query, but with options \fB-e\fR and \fB--and\fR: .IP-$ ugrep -wi -e amount --and '\d+(\.\d+)?' myfile.txt+$ ugrep -wi -e amount --and '\\d+(\\.\\d+)?' myfile.txt .PP List all Unicode words in a file: .IP@@ -1368,7 +1381,6 @@ recursive search, showing only matches with up to 10 characters of context before and after: .IP $ ugrep -o -C20 -R -n -k -tjs FIXME- .PP Find blocks of text between lines matching BEGIN and END by using a lazy quantifier `*?' to match only what is necessary and pattern `\\n' to match@@ -1457,13 +1469,14 @@ recursively below up to two levels deeper (3 levels total): .IP $ ugrep -3 -l -tRpm '' rpm/ .PP-Monitor the system log for bug reports and ungroup multiple matches on a line:+Monitor the system log for bug reports and ungroup multiple matches from+matching lines into separate lines: .IP $ tail -f /var/log/system.log | ugrep -u -i -w bug .PP Interactive fuzzy search with Boolean search queries: .IP-$ ugrep -Q -l -% -Z3 --sort=best+$ ugrep -Q -l -% -Z2 --sort=best .PP Display all words in a MacRoman-encoded file that has CR newlines: .IP@@ -1474,7 +1487,7 @@ Display options related to "fuzzy" searching: $ ugrep --help fuzzy .PP .SH COPYRIGHT-Copyright (c) 2021,2025 Robert A. van Engelen <engelen@acm.org>+Copyright (c) 2021-2025 Robert A. van Engelen <engelen@acm.org> .PP \fBugrep\fR is released under the BSD\-3 license. All parts of the software have reasonable copyright terms permitting free redistribution. This includes
@@ -209,6 +209,7 @@ extern std::vector<std::string> flag_file_type; extern std::vector<std::string> flag_file_extension; extern std::vector<std::string> flag_file_magic; extern std::vector<std::string> flag_filter_magic_label;+extern std::vector<std::string> flag_from; extern std::vector<std::string> flag_glob; extern std::vector<std::string> flag_iglob; extern std::vector<std::string> flag_include;
@@ -30,7 +30,7 @@ @file glob.cpp @brief gitignore-style pathname globbing @author Robert van Engelen - engelen@genivia.com-@copyright (c) 2019-2019, Robert van Engelen, Genivia Inc. All rights reserved.+@copyright (c) 2019-2025, Robert van Engelen, Genivia Inc. All rights reserved. @copyright (c) BSD-3 License - see LICENSE.txt */ @@ -45,7 +45,12 @@ // one for the last deep ** wildcard) // - linear time complexity in the length of the text for usual cases, with // worst-case quadratic time-// - performs case-insensitive matching when the ic flag is set to true+// - performs case-insensitive matching when the icase flag is set to true+// - the lead option matches the leading path part of the glob against the+// pathname, for example the glob foo/bar/baz matches pathname foo/bar+// - the path option matches the path part but not the basename of a pathname,+// for example, the glob foo/bar/baz matches pathname foo/bar/baz/file.txt+// and the glob ./ (or just /) matches pathname file.txt // // Pathnames are normalized by removing any leading ./ and / from the pathname //@@ -56,7 +61,7 @@ // [a-z] matches one character in the selected range of characters // [^a-z] matches one character not in the selected range of characters // [!a-z] same as [^a-z]-// / when at the begin of the glob matches if the pathname has no /+// / when at the start of the glob matches the working directory // **/ matches zero or more directories // /** when at the end of the glob matches everything after the / // \? matches a ? (or any character specified after the backslash)@@ -78,8 +83,8 @@ // a/** matches a/x, a/y, a/x/y but not a, b/x // a\?b matches a?b but not a, b, ab, axb, a/b -// check if we are on a windows OS-#if defined(__WIN32__) || defined(_WIN32) || defined(WIN32) || defined(__CYGWIN__) || defined(__MINGW32__) || defined(__MINGW64__) || defined(__BORLANDC__)+// check if we are natively compiling for a Windows OS (not Cygwin and not MinGW)+#if (defined(__WIN32__) || defined(_WIN32) || defined(WIN32) || defined(_WIN64) || defined(__BORLANDC__)) && !defined(__CYGWIN__) && !defined(__MINGW32__) && !defined(__MINGW64__) # define OS_WIN #endif @@ -93,10 +98,10 @@ #define PATHSEP '/' #endif -static int utf8(const char **s, bool ic = false);+static int utf8(const char **s, bool icase = false); -// match text against glob, return true or false, perform case-insensitive match if ic is true-static bool match(const char *text, const char *glob, bool ic)+// match text against glob, return true or false, perform case-insensitive match if icase=true, partial when part=true, path only when path=true+static bool match(const char *text, const char *glob, bool icase, bool lead, bool path) { // to iteratively backtrack on * const char *text1_backup = NULL;@@ -128,7 +133,7 @@ static bool match(const char *text, const char *glob, bool ic) text2_backup = text; glob2_backup = glob; if (*text != '/')- glob++;+ ++glob; continue; } @@ -143,13 +148,13 @@ static bool match(const char *text, const char *glob, bool ic) break; utf8(&text);- glob++;+ ++glob; continue; case '[': {- // match character class, ignoring case if ic is true- int chr = utf8(&text, ic), last = 0x10ffff;+ // match character class, ignoring case if icase is true+ int chr = utf8(&text, icase), last = 0x10ffff; // match anything except / if (chr == PATHSEP)@@ -160,10 +165,10 @@ static bool match(const char *text, const char *glob, bool ic) // inverted character class if (reverse)- glob++;- glob++;+ ++glob;+ ++glob; - int lc = chr, uc = (ic && lc >= 'a' && lc <= 'z' ? toupper(chr) : chr);+ int lc = chr, uc = (icase && lc >= 'a' && lc <= 'z' ? toupper(chr) : chr); if (lc == uc) {@@ -186,28 +191,37 @@ static bool match(const char *text, const char *glob, bool ic) break; if (*glob)- glob++;+ ++glob; continue; } + case '/':+ if (*text != PATHSEP)+ break;+ ++text;+ ++glob;+ continue;+ case '\\': // literal match \-escaped character- glob++;+ ++glob; // FALLTHROUGH default:-#ifdef OS_WIN- if ((ic ? tolower(*glob) != tolower(*text) : *glob != *text) && !(*glob == '/' && *text == '\\'))-#else- if (ic ? tolower(*glob) != tolower(*text) : *glob != *text)-#endif+ if (icase ?+ tolower(static_cast<unsigned char>(*glob)) != tolower(static_cast<unsigned char>(*text)) :+ *glob != *text) break; - text++;- glob++;+ ++text;+ ++glob; continue; } + // the path option matches the path up to but not including the basename+ if (path && *glob == '\0' && *text == PATHSEP && strchr(text + 1, PATHSEP) == NULL)+ return true;+ if (glob1_backup != NULL && *text1_backup != PATHSEP) { // backtrack on the last *, do not jump over /@@ -228,61 +242,67 @@ static bool match(const char *text, const char *glob, bool ic) } while (*glob == '*')- glob++;- return *glob == '\0';+ ++glob;+ return (*glob == '\0' && !path) || (*glob == '/' && lead); } -// pathname or basename glob matching, returns true or false, perform case-insensitive match if ic is true-bool glob_match(const char *pathname, const char *basename, const char *glob, bool ic)+// pathname or basename glob matching, returns true or false, perform case-insensitive match if icase is true+bool glob_match(const char *pathname, const char *basename, const char *glob, bool icase, bool lead, bool path) { // if pathname starts with ./ then skip this while (pathname[0] == '.' && pathname[1] == PATHSEP) pathname += 2;- // if pathname starts with / then skip it+ // if pathname starts with / then skip / while (pathname[0] == PATHSEP) ++pathname; // match pathname if glob contains a / or match the basename otherwise if (strchr(glob, '/') != NULL) {- // a leading ./ or / in the glob means globbing the pathname after removing the ./ or /+ // remove leading ./ or / if (glob[0] == '.' && glob[1] == '/') glob += 2; else if (glob[0] == '/') ++glob;- return match(pathname, glob, ic);++ if (*glob != '\0')+ return match(pathname, glob, icase, lead, path);++ // the path option matches the path up to but not including the basename, check the ./ glob case (glob is empty)+ return *pathname == '\0' || (path && strchr(pathname, PATHSEP) == NULL); } - return match(basename, glob, ic);+ // match basename, unless matching an empty path to the basename which always matches+ return path || match(basename, glob, icase, false, false); } -// return wide character of UTF-8 multi-byte sequence, return ASCII lower case if ic is true-static int utf8(const char **s, bool ic)+// return wide character of UTF-8 multi-byte sequence, return ASCII lower case if icase is true+static int utf8(const char **s, bool icase) {- int c1, c2, c3, c = (unsigned char)**s;+ int c1, c2, c3, c = static_cast<unsigned char>(**s); if (c != '\0')- (*s)++;+ ++*s; if (c < 0x80)- return ic ? tolower(c) : c;- c1 = (unsigned char)**s;+ return icase ? tolower(c) : c;+ c1 = static_cast<unsigned char>(**s); if (c < 0xC0 || (c == 0xC0 && c1 != 0x80) || c == 0xC1 || (c1 & 0xC0) != 0x80) return 0xFFFD; if (c1 != '\0')- (*s)++;+ ++*s; c1 &= 0x3F; if (c < 0xE0) return (((c & 0x1F) << 6) | c1);- c2 = (unsigned char)**s;+ c2 = static_cast<unsigned char>(**s); if ((c == 0xE0 && c1 < 0x20) || (c2 & 0xC0) != 0x80) return 0xFFFD; if (c2 != '\0')- (*s)++;+ ++*s; c2 &= 0x3F; if (c < 0xF0) return (((c & 0x0F) << 12) | (c1 << 6) | c2);- c3 = (unsigned char)**s;+ c3 = static_cast<unsigned char>(**s); if (c3 != '\0')- (*s)++;+ ++*s; if ((c == 0xF0 && c1 < 0x10) || (c == 0xF4 && c1 >= 0x10) || c >= 0xF5 || (c3 & 0xC0) != 0x80) return 0xFFFD; return (((c & 0x07) << 18) | (c1 << 12) | (c2 << 6) | (c3 & 0x3F));
@@ -35,4 +35,4 @@ */ // pathname or basename glob matching, returns true or false, perform case-insensitive match if ic is true-bool glob_match(const char *pathname, const char *basename, const char *glob, bool ic = false);+bool glob_match(const char *pathname, const char *basename, const char *glob, bool icase = false, bool lead = false, bool path = false);
@@ -537,7 +537,7 @@ void Output::binary_file_matches(const char *pathname, const std::string& partna return; str(color_off);- str("Binary file ", 12);+ str("Binary file "); str(color_fn); if (pathname != Static::LABEL_STANDARD_INPUT && color_hl != NULL)@@ -556,8 +556,7 @@ void Output::binary_file_matches(const char *pathname, const std::string& partna } str(color_off);-- str(" matches", 8);+ str(" matches"); nl(); mode_ |= BINARY;
@@ -733,13 +733,20 @@ class Output { mode_ |= FLUSH; } - // flush if output is line buffered to flush each line, unless we have to wait to acquire the output lock- inline void check_flush()+ // try to flush if we do not have to wait to acquire the output lock+ inline void try_flush() {- if (mode_ == FLUSH && (sync == NULL || sync->try_acquire(lock_)))+ if (sync == NULL || sync->try_acquire(lock_)) flush(); } + // flush if output is line buffered, unless we have to wait to acquire the output lock+ inline void check_flush()+ {+ if (mode_ == FLUSH)+ try_flush();+ }+ // hold the output and do not flush, buffer all output until further notice void hold() {@@ -784,6 +791,90 @@ class Output { sync->acquire(lock_, slot_); } + // discard buffered output+ void discard()+ {+ buf_ = buffers_.begin();+ cur_ = buf_->data;+ }++ // flush output and release sync slot, if one was assigned with sync_on()+ void release()+ {+ if ((mode_ & HOLD) == 0)+ flush();+ else+ discard();++ mode_ = flag_line_buffered ? FLUSH : 0;++ if (sync != NULL)+ sync->release(lock_);+ }++ // end output in ORDERED mode (--sort)+ void end()+ {+ if (sync != NULL)+ sync->finish(lock_, slot_);+ }++ // cancel output+ void cancel()+ {+ eof = true;++ if (sync != NULL)+ sync->cancel();+ }++ // true if output was cancelled()+ bool cancelled()+ {+ return sync != NULL && sync->cancelled();+ }++ // output color when set+ void color(const char *arg);++ // output the header part of the match, preceding the matched line+ void header(const char *pathname, const std::string& partname, bool& heading, size_t lineno, reflex::AbstractMatcher *matcher, size_t byte_offset, const char *sep, bool newline);++ // output the pathname header for --files_with_matches and --count+ void header(const char *pathname, const std::string& partname);++ // output "Binary file ... matches"+ void binary_file_matches(const char *pathname, const std::string& partname);++ // output format with option --format-begin and --format-end+ void format(const char *format, size_t matches);++ // output formatted match with options --format, --format-open, --format-close, returns false when nothing is output+ bool format(const char *format, const char *pathname, const std::string& partname, size_t matches, size_t *matching, reflex::AbstractMatcher *matcher, bool& heading, bool body, bool next);++ // output formatted inverted match with options -v --format, --format-open, --format-close+ void format_invert(const char *format, const char *pathname, const std::string& partname, size_t matches, size_t lineno, size_t offset, const char *ptr, size_t size, bool& heading, bool next);++ // output a quoted string with escapes for \ and "+ void quote(const char *data, size_t size);++ // output quoted string in C/C+++ void cpp(const char *data, size_t size);++ // output quoted string in CSV+ void csv(const char *data, size_t size);++ // output in hex+ void hex(const char *data, size_t size);++ // output quoted string in JSON+ void json(const char *data, size_t size);++ // output in XML+ void xml(const char *data, size_t size);++ protected:+ // acquire lock and flush the buffers, if not held back void flush() {@@ -880,90 +971,6 @@ class Output { // flush a block of data as truncated lines limited to --width columns bool flush_truncated_lines(const char *data, size_t size); - // discard buffered output- void discard()- {- buf_ = buffers_.begin();- cur_ = buf_->data;- }-- // flush output and release sync slot, if one was assigned with sync_on()- void release()- {- if ((mode_ & HOLD) == 0)- flush();- else- discard();-- mode_ = flag_line_buffered ? FLUSH : 0;-- if (sync != NULL)- sync->release(lock_);- }-- // end output in ORDERED mode (--sort)- void end()- {- if (sync != NULL)- sync->finish(lock_, slot_);- }-- // cancel output- void cancel()- {- eof = true;-- if (sync != NULL)- sync->cancel();- }-- // true if output was cancelled()- bool cancelled()- {- return sync != NULL && sync->cancelled();- }-- // output color when set- void color(const char *arg);-- // output the header part of the match, preceding the matched line- void header(const char *pathname, const std::string& partname, bool& heading, size_t lineno, reflex::AbstractMatcher *matcher, size_t byte_offset, const char *sep, bool newline);-- // output the pathname header for --files_with_matches and --count- void header(const char *pathname, const std::string& partname);-- // output "Binary file ... matches"- void binary_file_matches(const char *pathname, const std::string& partname);-- // output format with option --format-begin and --format-end- void format(const char *format, size_t matches);-- // output formatted match with options --format, --format-open, --format-close, returns false when nothing is output- bool format(const char *format, const char *pathname, const std::string& partname, size_t matches, size_t *matching, reflex::AbstractMatcher *matcher, bool& heading, bool body, bool next);-- // output formatted inverted match with options -v --format, --format-open, --format-close- void format_invert(const char *format, const char *pathname, const std::string& partname, size_t matches, size_t lineno, size_t offset, const char *ptr, size_t size, bool& heading, bool next);-- // output a quoted string with escapes for \ and "- void quote(const char *data, size_t size);-- // output quoted string in C/C++- void cpp(const char *data, size_t size);-- // output quoted string in CSV- void csv(const char *data, size_t size);-- // output in hex- void hex(const char *data, size_t size);-- // output quoted string in JSON- void json(const char *data, size_t size);-- // output in XML- void xml(const char *data, size_t size);-- protected:- // next buffer, allocate one if needed (when multi-threaded and lock is owned by another thread) void next() {
@@ -35,7 +35,7 @@ */ // DO NOT ALTER THIS LINE: updated by makemake.sh and we need it physically here for MSVC++ build from source-#define UGREP_VERSION "7.3.0"+#define UGREP_VERSION "7.4.0" // use a task-parallel thread to decompress the stream into a pipe to search, also handles nested archives #define WITH_DECOMPRESSION_THREAD@@ -44,7 +44,7 @@ #define WITH_SKIP_HIDDEN_ARCHIVES // check if we are compiling for a windows OS, but not Cygwin or MinGW-#if (defined(__WIN32__) || defined(_WIN32) || defined(WIN32) || defined(__BORLANDC__)) && !defined(__CYGWIN__) && !defined(__MINGW32__) && !defined(__MINGW64__)+#if (defined(__WIN32__) || defined(_WIN32) || defined(WIN32) || defined(_WIN64) || defined(__BORLANDC__)) && !defined(__CYGWIN__) && !defined(__MINGW32__) && !defined(__MINGW64__) # define OS_WIN #endif
@@ -1,5 +1,5 @@ /******************************************************************************\-* Copyright (c) 2019, 2025, Robert van Engelen, Genivia Inc. All rights reserved. *+* Copyright (c) 2019, Robert van Engelen, Genivia Inc. All rights reserved. * * * * Redistribution and use in source and binary forms, with or without * * modification, are permitted provided that the following conditions are met: *@@ -140,7 +140,6 @@ After this, you may want to test ugrep and install it (optional): #include <signal.h> #include <dirent.h>-#include <sys/select.h> #include <sys/stat.h> #include <unistd.h> @@ -443,6 +442,7 @@ std::vector<std::string> flag_file_type; std::vector<std::string> flag_file_extension; std::vector<std::string> flag_file_magic; std::vector<std::string> flag_filter_magic_label;+std::vector<std::string> flag_from; std::vector<std::string> flag_glob; std::vector<std::string> flag_iglob; std::vector<std::string> flag_include;@@ -686,7 +686,7 @@ inline bool getline(reflex::BufferedInput& input, std::string& line) line.erase(); - while ((ch = input.get()) != EOF && ch != '\n')+ while ((ch = input.get()) != EOF && ch != '\n' && ch != '\0') line.push_back(ch); if (!line.empty() && line.back() == '\r')@@ -695,71 +695,6 @@ inline bool getline(reflex::BufferedInput& input, std::string& line) return ch == EOF && line.empty(); } -// read a line from mmap memory, returns true when eof-inline bool getline(const char *& here, size_t& left)-{- // read line from mmap memory- if (left == 0)- return true;-- const char *s = static_cast<const char*>(memchr(here, '\n', left));- if (s == NULL)- s = here + left;- else- ++s;-- left -= s - here;- here = s;-- return false;-}--// read a line from mmap memory or from buffered input or from unbuffered input, returns true when eof-inline bool getline(const char *& here, size_t& left, reflex::BufferedInput& buffered_input, reflex::Input& input, std::string& line)-{- if (here != NULL)- {- // read line from mmap memory- if (left == 0)- return true;- const char *s = static_cast<const char*>(memchr(here, '\n', left));- if (s == NULL)- s = here + left;- else- ++s;- line.assign(here, s - here);- left -= s - here;- here = s;- return false;- }-- int ch;-- line.erase();-- if (buffered_input.assigned())- {- // read line from buffered input- while ((ch = buffered_input.get()) != EOF)- {- line.push_back(ch);- if (ch == '\n')- break;- }- return ch == EOF && line.empty();- }-- // read line from unbuffered input- while ((ch = input.get()) != EOF)- {- line.push_back(ch);- if (ch == '\n')- break;- }-- return ch == EOF && line.empty();-}- // return true if s[0..n-1] contains a \0 (NUL) or a non-displayable invalid UTF-8 encoding inline bool is_binary(const char *s, size_t n) {@@ -775,6 +710,12 @@ inline bool is_binary(const char *s, size_t n) return memchr(s, '\0', n) != NULL; } +// return true if -X or if -W and match s[0..n-1] is binary+inline bool check_binary(const char *s, size_t n)+{+ return (flag_hex || (flag_with_hex && is_binary(s, n)));+}+ // check if a file's inode is the current output file inline bool is_output(ino_t inode) {@@ -2074,6 +2015,7 @@ struct Grep { { return a.cost > b.cost || (a.cost == b.cost && a.pathname > b.pathname); }+ }; // a job in the job queue@@ -2104,6 +2046,7 @@ struct Grep { std::string pathname; uint16_t cost; size_t slot;+ }; #ifdef WITH_LOCK_FREE_JOB_QUEUE@@ -2202,6 +2145,7 @@ struct Grep { std::condition_variable queue_data; // cv to control the job queue std::condition_variable queue_full; // cv to control the job queue std::atomic_size_t todo; // number of jobs in the queue+ }; #else@@ -2313,51 +2257,44 @@ struct Grep { std::mutex queue_mutex; // job queue mutex std::condition_variable queue_work; // cv to control the job queue std::atomic_size_t todo; // number of jobs in the queue, atomic for job stealing+ }; #endif #ifndef OS_WIN - // extend the reflex::Input::Handler to handle nonblocking stdin from a TTY or from a slow pipe+ // extend the reflex::Input::Handler to handle nonblocking stdin from a character device (TTY) or from a pipe struct StdInHandler : public reflex::Input::Handler { - StdInHandler(Grep *grep)+ StdInHandler(Grep& grep) : grep(grep) { } - Grep *grep;+ Grep& grep; - int operator()(FILE *file)+ size_t operator()(FILE*, char *buf, size_t len) {- grep->out.flush();+ if (len == 0)+ {+ // could not read input data, but we aren't at EOF yet (i.e. non-blocking read returned zero) - while (true)+ // invoke buffer shift GrepHandler when set, to output the rest of a matching line and its context+ grep.matcher->handle();++ // flush output+ grep.out.try_flush();+ }+ else if (!flag_text && !flag_hex && !flag_with_hex && flag_encoding_type != reflex::Input::file_encoding::null_data && !grep.binfile) {- struct timeval tv;- fd_set rfds, efds;- FD_ZERO(&rfds);- FD_ZERO(&efds);- FD_SET(0, &rfds);- FD_SET(0, &efds);- tv.tv_sec = 1;- tv.tv_usec = 0;- int r = ::select(fileno(file) + 1, &rfds, NULL, &efds, &tv);- if (r < 0 && errno != EINTR)- return 0;- if (r > 0 && FD_ISSET(fileno(file), &efds))- return 0;- if (r > 0)- break;+ // simpler form of binary file detection without Unicode UTF-8 validation, like GNU grep+ grep.binfile = (memchr(buf, 0, len) != NULL); } - // clear EAGAIN and EINTR error on the nonblocking stdin- clearerr(file);-- // no error- return 1;+ return len; }+ }; #endif@@ -2365,41 +2302,22 @@ struct Grep { // extend the reflex::AbstractMatcher::Handler with a grep object reference and references to some of the grep::search locals struct GrepHandler : public reflex::AbstractMatcher::Handler { - GrepHandler(Grep& grep, const char *& pathname, size_t& lineno, bool& heading, bool& binfile, bool& hex, bool& binary, size_t& matches, bool& stop)+ GrepHandler(Grep& grep, const char *& pathname) : grep(grep),- pathname(pathname),- lineno(lineno),- heading(heading),- binfile(binfile),- hex(hex),- binary(binary),- matches(matches),- stop(stop)+ pathname(pathname) { } Grep& grep; // grep object- const char *& pathname; // grep::search argument pathname- size_t& lineno; // grep::search lineno local variable- bool& heading; // grep::search heading local variable- bool& binfile; // grep::search binfile local variable- bool& hex; // grep::search hex local variable- bool& binary; // grep::search binary local variable- size_t& matches; // grep::search matches local variable- bool& stop; // grep::search stop local variable-- // define a reflex::AbstractMatcher functor to handle buffer shifts by moving restline string from the buffer to a temporary string buffer+ const char *& pathname; // grep::search() argument pathname reference (pointer may be changed)++ // define a reflex::AbstractMatcher functor to handle buffer shifts virtual void operator()(reflex::AbstractMatcher&, const char*, size_t, size_t) override {- // if the rest of the line was not output yet, then save it in a temporary string buffer to output later- if (grep.restline_data != NULL)- {- grep.restline.assign(grep.restline_data, grep.restline_size);- grep.restline_data = grep.restline.c_str();- }+ grep.output_or_save_restline(false, flag_hex_after > 0); } - // get the start of the after/before context, if present+ // get the start of the after/before context lines, if present void begin_before(reflex::AbstractMatcher& matcher, const char *buf, size_t len, size_t num, const char *& ptr, size_t& size, size_t& offset) { ptr = NULL;@@ -2411,7 +2329,7 @@ struct Grep { return; size_t current = matcher.lineno();- size_t between = current - lineno;+ size_t between = current - grep.lineno; if (between > 1) {@@ -2435,11 +2353,11 @@ struct Grep { size = e - s; offset = s - buf + num; - ++lineno;+ ++grep.lineno; } } - // advance to the next before context, if present+ // advance to the next before context line, if present void next_before(const char *buf, size_t len, size_t num, const char *& ptr, size_t& size, size_t& offset) { if (ptr == NULL)@@ -2465,17 +2383,34 @@ struct Grep { size = eol - pos; offset = pos - buf + num; - ++lineno;+ ++grep.lineno; } }++ };++ // extend event GrepHandler for --only-matching with context+ struct OnlyMatchingContextGrepHandler : public GrepHandler {++ OnlyMatchingContextGrepHandler(Grep& grep, const char *& pathname)+ :+ GrepHandler(grep, pathname)+ { }++ // define a reflex::AbstractMatcher functor to handle buffer shifts by saving the restline used for context+ virtual void operator()(reflex::AbstractMatcher&, const char*, size_t, size_t) override+ {+ grep.save_restline();+ }+ }; // extend event GrepHandler to output invert match lines for -v struct InvertMatchGrepHandler : public GrepHandler { - InvertMatchGrepHandler(Grep& grep, const char *& pathname, size_t& lineno, bool& heading, bool& binfile, bool& hex, bool& binary, size_t& matches, bool& stop)+ InvertMatchGrepHandler(Grep& grep, const char *& pathname) :- GrepHandler(grep, pathname, lineno, heading, binfile, hex, binary, matches, stop)+ GrepHandler(grep, pathname) { } // functor invoked by the reflex::AbstractMatcher when the buffer contents are shifted out, also called explicitly in grep::search@@ -2485,48 +2420,52 @@ struct Grep { size_t size; size_t offset; + // if no gap to shift buffer, then let gap span the buffer until the begin of the matching line (bol)+ if (len == 0)+ len = matcher.before().len;+ begin_before(matcher, buf, len, num, ptr, size, offset); while (ptr != NULL) { // --range: max line exceeded?- if (flag_max_line > 0 && lineno > flag_max_line)+ if (flag_max_line > 0 && grep.lineno > flag_max_line) break; // --max-files: max reached?- if (matches == 0 && !Stats::found_part())+ if (grep.matches == 0 && !Stats::found_part()) {- stop = true;+ grep.stop = true; break; } // --max-count: max number of matches reached?- if (flag_max_count > 0 && matches >= flag_max_count)+ if (flag_max_count > 0 && grep.matches >= flag_max_count) break; // output blocked? if (grep.out.eof) break; - ++matches;+ ++grep.matches; if (flag_with_hex)- binary = false;+ grep.binary = false; - binary = binary || flag_hex || (flag_with_hex && is_binary(ptr, size));+ grep.binary = grep.binary || check_binary(ptr, size); - if (binfile || (binary && !flag_hex && !flag_with_hex))+ if (grep.binfile || (grep.binary && !flag_hex && !flag_with_hex)) break; - if (hex && !binary)+ if (grep.hex && !grep.binary) grep.out.dump.done(); if (!flag_no_header)- grep.out.header(pathname, grep.partname, heading, lineno, NULL, offset, flag_separator, binary);+ grep.out.header(pathname, grep.partname, grep.heading, grep.lineno, NULL, offset, flag_separator, grep.binary); - hex = binary;+ grep.hex = grep.binary; - if (binary)+ if (grep.binary) { grep.out.dump.hex(Output::Dump::HEX_LINE, offset, ptr, size); }@@ -2550,14 +2489,15 @@ struct Grep { next_before(buf, len, num, ptr, size, offset); } }+ }; // extend event GrepHandler to output formatted invert match lines for --format -v struct FormatInvertMatchGrepHandler : public GrepHandler { - FormatInvertMatchGrepHandler(Grep& grep, const char *& pathname, size_t& lineno, bool& heading, bool& binfile, bool& hex, bool& binary, size_t& matches, bool& stop)+ FormatInvertMatchGrepHandler(Grep& grep, const char *& pathname) :- GrepHandler(grep, pathname, lineno, heading, binfile, hex, binary, matches, stop)+ GrepHandler(grep, pathname) { } // functor invoked by the reflex::AbstractMatcher when the buffer contents are shifted out, also called explicitly in grep::search@@ -2567,16 +2507,20 @@ struct Grep { size_t size; size_t offset; + // if no gap to shift buffer, then let gap span the buffer until the begin of the matching line (bol)+ if (len == 0)+ len = matcher.before().len;+ begin_before(matcher, buf, len, num, ptr, size, offset); while (ptr != NULL) { // --range: max line exceeded?- if (flag_max_line > 0 && lineno > flag_max_line)+ if (flag_max_line > 0 && grep.lineno > flag_max_line) break; // output --format-open- if (matches == 0)+ if (grep.matches == 0) { // --format-open or --format-close: we must out.acquire() lock before Stats::found_part() if (flag_format_open != NULL || flag_format_close != NULL)@@ -2585,38 +2529,39 @@ struct Grep { // --max-files: max reached? if (!Stats::found_part()) {- stop = true;+ grep.stop = true; break; } if (flag_format_open != NULL)- grep.out.format(flag_format_open, pathname, grep.partname, Stats::found_parts(), NULL, &matcher, heading, false, Stats::found_parts() > 1);+ grep.out.format(flag_format_open, pathname, grep.partname, Stats::found_parts(), NULL, &matcher, grep.heading, false, Stats::found_parts() > 1); } // --max-count: max number of matches reached?- if (flag_max_count > 0 && matches >= flag_max_count)+ if (flag_max_count > 0 && grep.matches >= flag_max_count) break; // output blocked? if (grep.out.eof) break; - ++matches;+ ++grep.matches; // output --format- grep.out.format_invert(flag_format, pathname, grep.partname, matches, lineno, offset, ptr, size - (size > 0 && ptr[size - 1] == '\n'), heading, matches > 1);+ grep.out.format_invert(flag_format, pathname, grep.partname, grep.matches, grep.lineno, offset, ptr, size - (size > 0 && ptr[size - 1] == '\n'), grep.heading, grep.matches > 1); next_before(buf, len, num, ptr, size, offset); } }+ }; // extend event GrepHandler to output any context lines for -y struct AnyLineGrepHandler : public GrepHandler { - AnyLineGrepHandler(Grep& grep, const char *& pathname, size_t& lineno, bool& heading, bool& binfile, bool& hex, bool& binary, size_t& matches, bool& stop)+ AnyLineGrepHandler(Grep& grep, const char *& pathname) :- GrepHandler(grep, pathname, lineno, heading, binfile, hex, binary, matches, stop)+ GrepHandler(grep, pathname) { } // functor invoked by the reflex::AbstractMatcher when the buffer contents are shifted out, also called explicitly in grep::search@@ -2626,43 +2571,13 @@ struct Grep { size_t size; size_t offset; - begin_before(matcher, buf, len, num, ptr, size, offset);+ // if no gap to shift buffer, then let gap span the buffer until the begin of the matching line (bol)+ if (len == 0)+ len = matcher.before().len; - // output the rest of the matching line before the context lines- if (grep.restline_data != NULL)- {- if (lineno != matcher.lineno() || flag_ungroup)- {- if (binary)- {- grep.out.dump.hex(flag_invert_match ? Output::Dump::HEX_CONTEXT_LINE : Output::Dump::HEX_LINE, grep.restline_last, grep.restline_data, grep.restline_size);- grep.out.dump.done();- }- else- {- bool lf_only = false;- if (grep.restline_size > 0)- {- lf_only = grep.restline_data[grep.restline_size - 1] == '\n';- grep.restline_size -= lf_only;- if (grep.restline_size > 0)- {- grep.out.str(flag_invert_match ? color_cx : color_sl);- grep.out.str(grep.restline_data, grep.restline_size);- grep.out.str(color_off);- }- }- grep.out.nl(lf_only);- }+ begin_before(matcher, buf, len, num, ptr, size, offset); - grep.restline_data = NULL;- }- else- {- // save restline before the buffer shifts- GrepHandler::operator()(matcher, buf, len, num);- }- }+ grep.output_or_save_restline(flag_invert_match); // context colors with or without -v short v_hex_context_line = flag_invert_match ? Output::Dump::HEX_LINE : Output::Dump::HEX_CONTEXT_LINE;@@ -2672,23 +2587,23 @@ struct Grep { while (ptr != NULL) { // --range: max line exceeded?- if (flag_max_line > 0 && lineno > flag_max_line)+ if (flag_max_line > 0 && grep.lineno > flag_max_line) break; - if (matches == 0 && flag_invert_match)+ if (grep.matches == 0 && flag_invert_match) { // --max-files: max reached? if (!Stats::found_part()) {- stop = true;+ grep.stop = true; break; } } // --max-count: max number of matches reached?- if (flag_invert_match && flag_max_count > 0 && matches >= flag_max_count)+ if (flag_invert_match && flag_max_count > 0 && grep.matches >= flag_max_count) {- stop = true;+ grep.stop = true; break; } @@ -2697,25 +2612,25 @@ struct Grep { break; if (flag_with_hex)- binary = false;+ grep.binary = false; if (flag_invert_match)- ++matches;+ ++grep.matches; - binary = binary || flag_hex || (flag_with_hex && is_binary(ptr, size));+ grep.binary = grep.binary || check_binary(ptr, size); - if (binfile || (binary && !flag_hex && !flag_with_hex))+ if (grep.binfile || (grep.binary && !flag_hex && !flag_with_hex)) break; - if (hex && !binary)+ if (grep.hex && !grep.binary) grep.out.dump.done(); if (!flag_no_header)- grep.out.header(pathname, grep.partname, heading, lineno, NULL, offset, separator, binary);+ grep.out.header(pathname, grep.partname, grep.heading, grep.lineno, NULL, offset, separator, grep.binary); - hex = binary;+ grep.hex = grep.binary; - if (binary)+ if (grep.binary) { grep.out.dump.hex(v_hex_context_line, offset, ptr, size); }@@ -2739,6 +2654,7 @@ struct Grep { next_before(buf, len, num, ptr, size, offset); } }+ }; // extend event AnyLineGrepHandler to output specific context lines for -ABC without -v@@ -2784,9 +2700,9 @@ struct Grep { }; - ContextGrepHandler(Grep& grep, const char *& pathname, size_t& lineno, bool& heading, bool& binfile, bool& hex, bool& binary, size_t& matches, bool& stop)+ ContextGrepHandler(Grep& grep, const char *& pathname) :- AnyLineGrepHandler(grep, pathname, lineno, heading, binfile, hex, binary, matches, stop)+ AnyLineGrepHandler(grep, pathname) { } // output the before context@@ -2795,7 +2711,7 @@ struct Grep { // the group separator indicates lines skipped between contexts, like GNU grep if (state.after_lineno > 0 && state.after_lineno + state.after_length < grep.matcher->lineno() - state.before_length) {- if (hex)+ if (grep.hex) grep.out.dump.done(); if (flag_group_separator != NULL)@@ -2829,19 +2745,19 @@ struct Grep { size_t j = (state.before_index + i) % state.before_length; const ContextState::Line& before_line = state.before_lines[j]; - if (hex && !before_line.binary)+ if (grep.hex && !before_line.binary) grep.out.dump.done(); if (!flag_no_header)- grep.out.header(pathname, grep.partname, heading, before_lineno + i, NULL, before_line.offset, flag_separator_dash, before_line.binary);+ grep.out.header(pathname, grep.partname, grep.heading, before_lineno + i, NULL, before_line.offset, flag_separator_dash, before_line.binary); - hex = before_line.binary;+ grep.hex = before_line.binary; // lines are always saved at the moment, so ptr is NULL, but we may optimize this in the future to not always save const char *ptr = before_line.ptr != NULL ? before_line.ptr : before_line.line.c_str(); size_t size = before_line.size; - if (hex)+ if (grep.hex) { grep.out.dump.hex(Output::Dump::HEX_CONTEXT_LINE, before_line.offset, ptr, size); }@@ -2884,55 +2800,26 @@ struct Grep { size_t size; size_t offset; + // if no gap to shift buffer, then let gap span the buffer until the begin of the matching line (bol)+ if (len == 0)+ len = matcher.before().len;+ // if we only need the before context, then look for it right before the current lineno if (state.after_length >= flag_after_context) { size_t current = matcher.lineno();- if (lineno + flag_before_context + 1 < current)- lineno = current - flag_before_context - 1;+ if (grep.lineno + flag_before_context + 1 < current)+ grep.lineno = current - flag_before_context - 1; } begin_before(matcher, buf, len, num, ptr, size, offset); - // output the rest of the matching line before the context lines- if (grep.restline_data != NULL)- {- if (lineno != matcher.lineno() || flag_ungroup)- {- if (binary)- {- grep.out.dump.hex(Output::Dump::HEX_LINE, grep.restline_last, grep.restline_data, grep.restline_size);- }- else- {- bool lf_only = false;- if (grep.restline_size > 0)- {- lf_only = grep.restline_data[grep.restline_size - 1] == '\n';- grep.restline_size -= lf_only;- if (grep.restline_size > 0)- {- grep.out.str(color_sl);- grep.out.str(grep.restline_data, grep.restline_size);- grep.out.str(color_off);- }- }- grep.out.nl(lf_only);- }-- grep.restline_data = NULL;- }- else- {- // save restline before the buffer shifts- GrepHandler::operator()(matcher, buf, len, num);- }- }+ grep.output_or_save_restline(); while (ptr != NULL) { // --range: max line exceeded?- if (flag_max_line > 0 && lineno > flag_max_line)+ if (flag_max_line > 0 && grep.lineno > flag_max_line) break; // output blocked?@@ -2940,26 +2827,26 @@ struct Grep { break; if (flag_with_hex)- binary = false;+ grep.binary = false; - binary = binary || flag_hex || (flag_with_hex && is_binary(ptr, size));+ grep.binary = grep.binary || check_binary(ptr, size); - if (binfile || (binary && !flag_hex && !flag_with_hex))+ if (grep.binfile || (grep.binary && !flag_hex && !flag_with_hex)) break; if (state.after_lineno > 0 && state.after_length < flag_after_context) { ++state.after_length; - if (hex && !binary)+ if (grep.hex && !grep.binary) grep.out.dump.done(); if (!flag_no_header)- grep.out.header(pathname, grep.partname, heading, lineno, NULL, offset, flag_separator_dash, binary);+ grep.out.header(pathname, grep.partname, grep.heading, grep.lineno, NULL, offset, flag_separator_dash, grep.binary); - hex = binary;+ grep.hex = grep.binary; - if (binary)+ if (grep.binary) { grep.out.dump.hex(Output::Dump::HEX_CONTEXT_LINE, offset, ptr, size); }@@ -2986,7 +2873,7 @@ struct Grep { ++state.before_length; state.before_index %= state.before_length;- state.before_lines[state.before_index].binary = binary;+ state.before_lines[state.before_index].binary = grep.binary; state.before_lines[state.before_index].offset = offset; state.before_lines[state.before_index].ptr = ptr; state.before_lines[state.before_index].size = size;@@ -3074,18 +2961,18 @@ struct Grep { }; - InvertContextGrepHandler(Grep& grep, const char *& pathname, size_t& lineno, bool& heading, bool& binfile, bool& hex, bool& binary, size_t& matches, bool& stop)+ InvertContextGrepHandler(Grep& grep, const char *& pathname) :- AnyLineGrepHandler(grep, pathname, lineno, heading, binfile, hex, binary, matches, stop)+ AnyLineGrepHandler(grep, pathname) { } // output the inverted before context void output_before_context() { // the group separator indicates lines skipped between contexts, like GNU grep- if (state.after_lineno > 0 && state.after_lineno + flag_after_context + flag_before_context < lineno && flag_group_separator != NULL)+ if (state.after_lineno > 0 && state.after_lineno + flag_after_context + flag_before_context < grep.lineno && flag_group_separator != NULL) {- if (hex)+ if (grep.hex) grep.out.dump.done(); if (flag_query && !flag_text)@@ -3109,7 +2996,7 @@ struct Grep { if (state.before_length > 0) { // the first line number of the before context- size_t before_lineno = lineno - state.before_length;+ size_t before_lineno = grep.lineno - state.before_length; for (size_t i = 0; i < state.before_length; ++i) {@@ -3117,13 +3004,13 @@ struct Grep { const InvertContextState::Line& before_line = state.before_lines[j]; size_t offset = before_line.matches.empty() ? before_line.offset : before_line.matches.front().offset; - if (hex && !before_line.binary)+ if (grep.hex && !before_line.binary) grep.out.dump.done(); if (!flag_no_header)- grep.out.header(pathname, grep.partname, heading, before_lineno + i, NULL, offset, flag_separator_dash, before_line.binary);+ grep.out.header(pathname, grep.partname, grep.heading, before_lineno + i, NULL, offset, flag_separator_dash, before_line.binary); - hex = before_line.binary;+ grep.hex = before_line.binary; const char *ptr = before_line.line.c_str(); size_t size = before_line.line.size();@@ -3131,7 +3018,7 @@ struct Grep { for (const auto& match : before_line.matches) {- if (hex)+ if (grep.hex) { grep.out.dump.hex(Output::Dump::HEX_CONTEXT_LINE, match.offset - (match.pos - pos), ptr + pos, match.pos - pos); grep.out.dump.hex(Output::Dump::HEX_CONTEXT_MATCH, match.offset, ptr + match.pos, match.size);@@ -3160,7 +3047,7 @@ struct Grep { pos = match.pos + match.size; } - if (hex)+ if (grep.hex) { grep.out.dump.hex(Output::Dump::HEX_CONTEXT_LINE, before_line.offset + pos, ptr + pos, size - pos); }@@ -3186,7 +3073,7 @@ struct Grep { // reset the context state state.before_index = 0; state.before_length = 0;- state.after_lineno = lineno + 1;+ state.after_lineno = grep.lineno + 1; } // add context line with the first match to the inverted before context@@ -3197,7 +3084,7 @@ struct Grep { state.before_index %= state.before_length; - state.before_lines[state.before_index].binary = binary;+ state.before_lines[state.before_index].binary = grep.binary; state.before_lines[state.before_index].columno = columno; state.before_lines[state.before_index].offset = offset; state.before_lines[state.before_index].line.assign(bol, eol - bol);@@ -3224,91 +3111,62 @@ struct Grep { size_t size; size_t offset; - begin_before(matcher, buf, len, num, ptr, size, offset);+ // if no gap to shift buffer, then let gap span the buffer until the begin of the matching line (bol)+ if (len == 0)+ len = matcher.before().len; - // output the rest of the "after" matching line- if (grep.restline_data != NULL)- {- if (lineno != matcher.lineno() || flag_ungroup)- {- if (binary)- {- grep.out.dump.hex(Output::Dump::HEX_CONTEXT_LINE, grep.restline_last, grep.restline_data, grep.restline_size);- }- else- {- bool lf_only = false;- if (grep.restline_size > 0)- {- lf_only = grep.restline_data[grep.restline_size - 1] == '\n';- grep.restline_size -= lf_only;- if (grep.restline_size > 0)- {- grep.out.str(color_cx);- grep.out.str(grep.restline_data, grep.restline_size);- grep.out.str(color_off);- }- }- grep.out.nl(lf_only);- }+ begin_before(matcher, buf, len, num, ptr, size, offset); - grep.restline_data = NULL;- }- else- {- // save restline before the buffer shifts- GrepHandler::operator()(matcher, buf, len, num);- }- }+ grep.output_or_save_restline(true); if (ptr != NULL) output_before_context(); while (ptr != NULL) {- state.after_lineno = lineno + 1;+ state.after_lineno = grep.lineno + 1; // --range: max line exceeded?- if (flag_max_line > 0 && lineno > flag_max_line)+ if (flag_max_line > 0 && grep.lineno > flag_max_line) break; - if (matches == 0)+ if (grep.matches == 0) { // --max-files: max reached? if (!Stats::found_part()) {- stop = true;+ grep.stop = true; break; } } // --max-count: max number of matches reached?- if (flag_max_count > 0 && matches >= flag_max_count)+ if (flag_max_count > 0 && grep.matches >= flag_max_count) break; // output blocked? if (grep.out.eof) break; - ++matches;+ ++grep.matches; if (flag_with_hex)- binary = false;+ grep.binary = false; - binary = binary || flag_hex || (flag_with_hex && is_binary(ptr, size));+ grep.binary = grep.binary || check_binary(ptr, size); - if (binfile || (binary && !flag_hex && !flag_with_hex))+ if (grep.binfile || (grep.binary && !flag_hex && !flag_with_hex)) break; - if (hex && !binary)+ if (grep.hex && !grep.binary) grep.out.dump.done(); if (!flag_no_header)- grep.out.header(pathname, grep.partname, heading, lineno, NULL, offset, flag_separator, binary);+ grep.out.header(pathname, grep.partname, grep.heading, grep.lineno, NULL, offset, flag_separator, grep.binary); - hex = binary;+ grep.hex = grep.binary; - if (binary)+ if (grep.binary) { grep.out.dump.hex(Output::Dump::HEX_LINE, offset, ptr, size); }@@ -3348,7 +3206,7 @@ struct Grep { matchers(matchers), file_in(NULL) #ifndef OS_WIN- , stdin_handler(this)+ , stdin_handler(*this) #endif #ifdef HAVE_LIBZ #ifdef WITH_DECOMPRESSION_THREAD@@ -3600,8 +3458,6 @@ struct Grep { int fl = fcntl(fd, F_GETFL); if (fl >= 0) fcntl(fd, F_SETFL, fl | O_NONBLOCK);- else- clearerr(file_in); } } #endif@@ -4036,41 +3892,16 @@ struct Grep { #endif #endif -#ifdef WITH_STDIN_DRAIN-#ifndef OS_WIN- // drain stdin until eof- if (file_in == stdin && !feof(stdin))+#if defined(WITH_STDIN_DRAIN) && !defined(OS_WIN)++ // drain stdin when non-seekable file such as a pipe until eof+ if (file_in == stdin && !feof(stdin) && fseek(stdin, 0, SEEK_END) < 0 && errno != EINVAL) {- if (fseek(stdin, 0, SEEK_END) < 0 && errno != EINVAL)- {- char buf[16384];- while (true)- {- if (fread(buf, 1, sizeof(buf), stdin) == sizeof(buf))- continue;- if (feof(stdin))- break;- if (errno != EINTR)- break;- if (!(fcntl(0, F_GETFL) & O_NONBLOCK))- break;- struct timeval tv;- fd_set rfds, efds;- FD_ZERO(&rfds);- FD_ZERO(&efds);- FD_SET(0, &rfds);- FD_SET(0, &efds);- tv.tv_sec = 1;- tv.tv_usec = 0;- int r = ::select(1, &rfds, NULL, &efds, &tv);- if (r < 0 && errno != EINTR)- break;- if (r > 0 && FD_ISSET(0, &efds))- break;- }- }+ char buf[16384];+ while (input.get(buf, sizeof(buf)) > 0)+ continue; }-#endif+ #endif // close the current input file@@ -4088,6 +3919,16 @@ struct Grep { // specify input to read for matcher, when input is a regular file then try mmap for zero copy overhead bool init_read() {+ // initialize Grep search and output variables+ interactive = false;+ binfile = false;+ lineno = flag_min_line > 0 ? flag_min_line - 1 : 0;+ heading = flag_with_filename;+ hex = false;+ binary = false;+ matches = 0;+ stop = false;+ const char *base; size_t size; @@ -4105,30 +3946,45 @@ struct Grep { #if !defined(HAVE_PCRE2) && defined(HAVE_BOOST_REGEX) // buffer all input to work around Boost.Regex partial matching bug, but this may throw std::bad_alloc if the file is too large if (flag_perl_regexp)+ { matcher->buffer();+ }+ else #endif #ifndef OS_WIN- if (input == stdin)+ if (input == stdin && !flag_quiet && !flag_files_with_matches && !flag_count) { struct stat buf;- bool interactive = fstat(0, &buf) == 0 && (S_ISCHR(buf.st_mode) || S_ISFIFO(buf.st_mode)); - // if input is a TTY or pipe, then make stdin nonblocking and register a stdin handler to continue reading and to flush results to output- if (interactive)+ // if input is a character device (TTY) or a pipe, then make stdin nonblocking and register a stdin handler to continue reading and flush results to output+ if (fstat(0, &buf) == 0 && (S_ISCHR(buf.st_mode) || S_ISFIFO(buf.st_mode))) {- if (fcntl(0, F_SETFL, fcntl(0, F_GETFL) | O_NONBLOCK) == -1)- clearerr(stdin);- else+ interactive = (fcntl(0, F_SETFL, fcntl(0, F_GETFL) | O_NONBLOCK) != -1);+ if (interactive) matcher->in.set_handler(&stdin_handler); } }+#else+ { } #endif } - // -I: do not match binary- if (flag_binary_without_match && init_is_binary())- return false;+ // when non-blocking do not check init_is_binary(), wait until handler detects binary files+ if (!interactive)+ {+ if (flag_binary_without_match)+ {+ // -I: do not match binary files+ if (init_is_binary())+ return false;+ }+ else if (!flag_quiet && !flag_files_with_matches && !flag_count && flag_format == NULL && !flag_text && !flag_hex && !flag_with_hex)+ {+ // not -q, -l, -c, --format, -a -X, -W: check if initial part of the file is binary+ binfile = init_is_binary();+ }+ } // --range=[MIN,][MAX]: start searching at line MIN for (size_t i = flag_min_line; i > 1; --i)@@ -4138,11 +3994,11 @@ struct Grep { return true; } - // after opening a file with init_read, check if its initial part is binary+ // initial part of a file is binary bool init_is_binary() {- // check up to 64K in buffer for binary data, the buffer is a window over the input file- size_t avail = std::min<size_t>(matcher->avail(), 65536);+ // check buffer for binary data, the buffer is a 256K window over the input file+ size_t avail = matcher->avail(); if (avail == 0) return false; @@ -4160,6 +4016,66 @@ struct Grep { return is_binary(matcher->begin(), avail); } + // output the rest of the matching line or save it for later, uses lineno of previous match to check+ void output_or_save_restline(bool invert = false, bool hex_context = false)+ {+ if (restline_data != NULL)+ {+ if (flag_ungroup || lineno != matcher->lineno())+ output_restline(invert, hex_context);+ else+ save_restline();+ }+ }++ // output the rest of the matching line or save it for later, this is necessary when the buffer shifts+ void output_restline(bool invert = false, bool hex_context = false)+ {+ if (restline_data != NULL)+ {+ if (binary)+ {+ // --hexdump with ABC context: if binary then show hex context+ if (hex_context)+ {+ size_t right = flag_hex_after * flag_hex_columns - ((restline_last - 1) % flag_hex_columns) - 1;+ if (restline_last + right > matcher->first())+ right = matcher->first() - restline_last;+ if (right < restline_size)+ restline_size = right;+ }++ out.dump.hex(invert ? Output::Dump::HEX_CONTEXT_LINE : Output::Dump::HEX_LINE, restline_last, restline_data, restline_size);+ }+ else+ {+ bool lf_only = false;+ if (restline_size > 0)+ {+ lf_only = restline_data[restline_size - 1] == '\n';+ restline_size -= lf_only;+ if (restline_size > 0)+ {+ out.str(invert ? color_cx : color_sl);+ out.str(restline_data, restline_size);+ out.str(color_off);+ }+ }+ out.nl(lf_only);+ }++ restline_data = NULL;+ }+ }++ // save the rest of the line to a string buffer to output later, may self-assign, so swap with a temp string+ void save_restline()+ {+ std::string temp(restline_data, restline_size);+ temp.swap(restline);+ restline_data = restline.c_str();+ }+ const char *filename; // the name of the file being searched std::string partname; // the name of an extracted file from an archive std::string restline; // a buffer to store the rest of a line to search@@ -4174,6 +4090,14 @@ struct Grep { MMap mmap; // mmap state reflex::Input input; // input to the matcher FILE *file_in; // the current input file+ bool interactive; // search non-blocking standard input (e.g. TTY or pipe)+ bool binfile; // the file being searched is a binary file+ size_t lineno; // the line number of the last match+ bool heading; // heading on/off, initially set to --with-filename+ bool hex; // the match is output in hex+ bool binary; // the match is binary as per option -X or -W+ size_t matches; // number of matches+ bool stop; // stop searching when --max-files max reached #ifndef OS_WIN StdInHandler stdin_handler; // a handler to handle nonblocking stdin from a TTY or a slow pipe #endif@@ -5479,6 +5403,10 @@ void options(std::list<std::pair<CNF::PATTERN,const char*>>& pattern_args, int a flag_format_end = strarg(arg + 11); else if (strncmp(arg, "format-open=", 12) == 0) flag_format_open = strarg(arg + 12);+ else if (strcmp(arg, "from") == 0) // legacy form --from FILE+ flag_from.emplace_back(getloptarg(argc, argv, "", i));+ else if (strncmp(arg, "from=", 5) == 0)+ flag_from.emplace_back(getloptarg(argc, argv, arg + 5, i)); else if (strcmp(arg, "fuzzy") == 0) flag_fuzzy = 1; else if (strncmp(arg, "fuzzy=", 6) == 0)@@ -6884,25 +6812,41 @@ void init(int argc, const char **argv) if (strcmp(flag_index, "safe") != 0 && strcmp(flag_index, "fast") != 0 && strcmp(flag_index, "log") != 0) usage("invalid argument --index=MODE, valid arguments are 'safe', 'fast' and 'log'"); - if (!flag_stdin && Static::arg_files.empty())+ // --from=FILE: add file pathnames to search from FILE as if specified on the command line+ for (const auto& from : flag_from) {- // if no FILE specified when reading standard input from a TTY then enable -R if not already -r or -R- if (isatty(STDIN_FILENO) && (flag_directories_action == Action::UNSP || flag_directories_action == Action::RECURSE))+ if (!from.empty()) {- if (flag_directories_action == Action::UNSP)- flag_directories_action = Action::RECURSE;+ FILE *file = NULL; - // recursive search with worker threads- flag_all_threads = true;- }- else- {- // if no FILE specified then read standard input- flag_stdin = true;+ if (fopen_smart(&file, from.c_str(), "r") != 0)+ error("option --from: cannot read", from.c_str());++ // read globs from the specified file or files+ reflex::BufferedInput input(file);+ std::string line;++ while (true)+ {+ // read the next line+ if (getline(input, line))+ break;++ if (!line.empty())+ {+ // push file pathnames imported from the file to the back of the arg files vector, saving std::string first in a list+ arg_strings.emplace_back();+ arg_strings.back().swap(line);+ Static::arg_files.push_back(arg_strings.back().c_str());+ }+ }++ if (file != stdin)+ fclose(file); } } - // check FILE arguments, warn about non-existing and non-readable files and directories+ // check all FILE arguments, warn about non-existing and non-readable files and directories auto file = Static::arg_files.begin(); while (file != Static::arg_files.end()) {@@ -6994,16 +6938,34 @@ void init(int argc, const char **argv) #endif } - // normalize --cpp, --csv, --json, --xml to their corresponding --format- if (flag_cpp)- {- flag_format_begin = "const struct grep {\n const char *file;\n size_t line;\n size_t column;\n size_t offset;\n const char *match;\n} matches[] = {\n";- flag_format_open = " // %f\n";- flag_format = " { %h, %n, %k, %b, %C },\n%u";- flag_format_close = " \n";- flag_format_end = " { NULL, 0, 0, 0, NULL }\n};\n";- }- else if (flag_csv)+ if (!flag_stdin && Static::arg_files.empty())+ {+ // if no FILE specified when reading standard input from a TTY then enable -R if not already -r or -R+ if (isatty(STDIN_FILENO) && (flag_directories_action == Action::UNSP || flag_directories_action == Action::RECURSE))+ {+ if (flag_directories_action == Action::UNSP)+ flag_directories_action = Action::RECURSE;++ // recursive search with worker threads+ flag_all_threads = true;+ }+ else+ {+ // if no FILE specified then read standard input+ flag_stdin = true;+ }+ }++ // normalize --cpp, --csv, --json, --xml to their corresponding --format+ if (flag_cpp)+ {+ flag_format_begin = "const struct grep {\n const char *file;\n size_t line;\n size_t column;\n size_t offset;\n const char *match;\n} matches[] = {\n";+ flag_format_open = " // %f\n";+ flag_format = " { %h, %n, %k, %b, %C },\n%u";+ flag_format_close = " \n";+ flag_format_end = " { NULL, 0, 0, 0, NULL }\n};\n";+ }+ else if (flag_csv) { flag_format_open = "%+"; flag_format = "%[,]$%H%N%K%B%V\n%u";@@ -7081,7 +7043,7 @@ void init(int argc, const char **argv) if (size > 0) {- std::string mount(mounts.substr(from, size));+ std::string mount(mounts, from, size); if (stat_fs(mount.c_str(), &buf) == 0) exclude_fs_ids.insert(fsid_to_uint64(buf.f_fsid));@@ -7119,7 +7081,7 @@ void init(int argc, const char **argv) if (size > 0) {- std::string mount(mounts.substr(from, size));+ std::string mount(mounts, from, size); if (stat_fs(mount.c_str(), &buf) == 0) include_fs_ids.insert(fsid_to_uint64(buf.f_fsid));@@ -7137,7 +7099,7 @@ void init(int argc, const char **argv) #endif - // --exclude-from: add globs to the exclude and exclude-dir lists+ // --exclude-from=FILE: add globs to the exclude and exclude-dir lists for (const auto& from : flag_exclude_from) { if (!from.empty())@@ -7155,7 +7117,7 @@ void init(int argc, const char **argv) } } - // --include-from: add globs to the include and include-dir lists+ // --include-from=FILE: add globs to the include and include-dir lists for (const auto& from : flag_include_from) { if (!from.empty())@@ -7193,7 +7155,7 @@ void init(int argc, const char **argv) --size; } - std::string type(types.substr(from, size));+ std::string type(types, from, size); size_t i = 0; bool found = false;@@ -7319,18 +7281,21 @@ void init(int argc, const char **argv) size_t size = to - from; - if (size == 0)- break;+ if (size > 0)+ {+ bool negate = size > 1 && (extensions[from] == '!' || extensions[from] == '^'); - bool negate = size > 1 && (extensions[from] == '!' || extensions[from] == '^');+ if (negate)+ {+ ++from;+ --size;+ } - if (negate)- {- ++from;- --size;+ flag_glob.emplace_back(glob.assign(negate ? "^*." : "*.").append(extensions, from, size)); } - flag_glob.emplace_back(glob.assign(negate ? "^*." : "*.").append(extensions.substr(from, size)));+ if (to >= extensions.size())+ break; from = to + 1; }@@ -7838,14 +7803,10 @@ void ugrep() Output::Tree::depth = 0; // clear internal combined all-include and all-exclude globs- flag_all_include.clear(); flag_all_exclude.clear();-- // all globs are case-sensitive by default, unless --glob-ignore-case and --iglob are specified- flag_include_iglob_size = 0;- flag_exclude_iglob_size = 0;- flag_include_iglob_dir_size = 0;- flag_exclude_iglob_dir_size = 0;+ flag_all_include.clear();+ flag_all_exclude_dir.clear();+ flag_all_include_dir.clear(); // --iglob: add case-insensitive globs to all-include-iglob/all-exclude-iglob if not --glob-ignore-case for (const auto& globs : flag_iglob)@@ -7867,34 +7828,41 @@ void ugrep() size_t size = to - from; - if (size == 0)- break;-- bool negate = size > 1 && (globs[from] == '!' || globs[from] == '^');-- if (negate)- {- ++from;- --size;- flag_all_exclude.emplace_back(globs.substr(from, size));- if (globs[from + size - 1] == '/')- ++flag_exclude_iglob_dir_size;- else- ++flag_exclude_iglob_size;- }- else+ if (size > 0) {- flag_all_include.emplace_back(globs.substr(from, size));- if (globs[from + size - 1] == '/')- ++flag_include_iglob_dir_size;+ bool negate = size > 1 && (globs[from] == '!' || globs[from] == '^');++ if (negate)+ {+ ++from;+ --size;+ if (globs[from + size - 1] == '/')+ flag_all_exclude_dir.emplace_back(globs, from, size - 1);+ else+ flag_all_exclude.emplace_back(globs, from, size);+ } else- ++flag_include_iglob_size;+ {+ if (globs[from + size - 1] == '/')+ flag_all_include_dir.emplace_back(globs, from, size - 1);+ else+ flag_all_include.emplace_back(globs, from, size);+ } } + if (to >= globs.size())+ break;+ from = to + 1; } } + // all --iglob globs are case-sensitive+ flag_include_iglob_size = flag_all_include.size();+ flag_exclude_iglob_size = flag_all_exclude.size();+ flag_include_iglob_dir_size = flag_all_include_dir.size();+ flag_exclude_iglob_dir_size = flag_all_exclude_dir.size();+ // -g, --glob: add globs to all-include/all-exclude for (const auto& globs : flag_glob) {@@ -7915,83 +7883,170 @@ void ugrep() size_t size = to - from; - if (size == 0)- break;-- bool negate = size > 1 && (globs[from] == '!' || globs[from] == '^');-- if (negate)- {- ++from;- --size;- flag_all_exclude.emplace_back(globs.substr(from, size));- }- else+ if (size > 0) {- flag_all_include.emplace_back(globs.substr(from, size));+ bool negate = size > 1 && (globs[from] == '!' || globs[from] == '^');++ if (negate)+ {+ ++from;+ --size;+ if (globs[from + size - 1] == '/')+ flag_all_exclude_dir.emplace_back(globs, from, size - 1);+ else+ flag_all_exclude.emplace_back(globs, from, size);+ }+ else+ {+ if (globs[from + size - 1] == '/')+ flag_all_include_dir.emplace_back(globs, from, size - 1);+ else+ flag_all_include.emplace_back(globs, from, size);+ } } + if (to >= globs.size())+ break;+ from = to + 1; } } - // populate the internal combined all-include and all-exclude globs- flag_all_include.insert(flag_all_include.begin(), flag_include.begin(), flag_include.end());- flag_all_exclude.insert(flag_all_exclude.begin(), flag_exclude.begin(), flag_exclude.end());- flag_all_include_dir = flag_include_dir;- flag_all_exclude_dir = flag_exclude_dir;+ // populate the internal combined all-include and all-exclude globs by appending --include and --exclude+ flag_all_exclude.insert(flag_all_exclude.end(), flag_exclude.begin(), flag_exclude.end());+ flag_all_include.insert(flag_all_include.end(), flag_include.begin(), flag_include.end());+ // same for dirs+ flag_all_exclude_dir.insert(flag_all_exclude_dir.end(), flag_exclude_dir.begin(), flag_exclude_dir.end());+ flag_all_include_dir.insert(flag_all_include_dir.end(), flag_include_dir.begin(), flag_include_dir.end()); - // all excluded files: normalize by moving directory globs (globs ending in a path separator /) to --exclude-dir+ // all excluded files: normalize by moving directory globs (globs ending in a path separator /) to --exclude-dir and remove consecutive duplicates+ std::string *last = NULL; auto i = flag_all_exclude.begin(); while (i != flag_all_exclude.end()) {- if (i->empty())+ // remove empty and duplicate --exclude globs+ if (i->empty() || (last != NULL && i->compare(*last) == 0)) { i = flag_all_exclude.erase(i);+ if (static_cast<size_t>(std::distance(flag_all_exclude.begin(), i)) < flag_exclude_iglob_size)+ --flag_exclude_iglob_size; } else if (i->back() == '/') {+ // move directory glob (ending in a /) from --exclude to --exclude-dir (these are all case sensitive) flag_all_exclude_dir.emplace_back(*i); i = flag_all_exclude.erase(i); } else {- ++i;+ last = &*i++; } } - // all included files: normalize by moving directory globs (globs ending in a path separator /) to --include-dir+ // if all --include files have paths, then add them to --include-dir to speed up searching+ std::vector<std::string> all_include_dir_from_iglob;+ std::vector<std::string> all_include_dir_from_glob;++ // all included files: normalize by moving or adding directory globs (globs ending in a path separator /) to --include-dir and remove consecutive duplicates+ bool add_wdir = false;+ last = NULL; i = flag_all_include.begin(); while (i != flag_all_include.end()) {- if (i->empty())+ // remove empty and duplicate --include globs+ if (i->empty() || (last != NULL && i->compare(*last) == 0)) { i = flag_all_include.erase(i);+ if (static_cast<size_t>(std::distance(flag_all_include.begin(), i)) < flag_include_iglob_size)+ --flag_include_iglob_size; } else { if (i->back() == '/') {+ // move directory glob (ending in a /) from --include to --include-dir (these are all case sensitive) flag_all_include_dir.emplace_back(*i); i = flag_all_include.erase(i); } else {- // if an include file glob starts with a dot, then enable searching hidden files and directories- if (i->front() == '.' || i->find(PATHSEPSTR ".") != std::string::npos)+ size_t last_sep = i->find_last_of('/');++ // if include file has a path, then add the path part to the include-dirs, respect case (in)sensitivity+ if (last_sep != std::string::npos)+ {+ size_t first_sep = i->find_first_of('/');++ if (first_sep == last_sep && (first_sep == 0 || (first_sep == 1 && i->front() == '.')))+ {+ add_wdir = true;+ }+ else if (static_cast<size_t>(std::distance(flag_all_include.begin(), i)) < flag_include_iglob_size)+ {+ all_include_dir_from_iglob.emplace_back(*i, 0, last_sep);+ if (first_sep == last_sep)+ all_include_dir_from_iglob.back().insert(0, "./");+ }+ else+ {+ all_include_dir_from_glob.emplace_back(*i, 0, last_sep);+ if (first_sep == last_sep)+ all_include_dir_from_glob.back().insert(0, "./");+ }+ }++ // if an include file glob starts with a dot, then enable --hidden+ if (last_sep != std::string::npos ? i->at(last_sep + 1) == '.' : i->front() == '.') flag_hidden = true; - ++i;+ last = &*i++; } } }+ + // update include-dir to search case insensitive directories collected from --iglob+ flag_include_iglob_dir_size += all_include_dir_from_iglob.size();+ flag_all_include_dir.insert(flag_all_include_dir.begin(), all_include_dir_from_iglob.begin(), all_include_dir_from_iglob.end()); - // if an include dir glob starts with a dot, then enable searching hidden files and directories+ // search the working dir?+ if (add_wdir)+ flag_all_include_dir.emplace_back("./");++ // update include-dir to search case sensitive directories collected from --include, --glob+ flag_all_include_dir.insert(flag_all_include_dir.end(), all_include_dir_from_glob.begin(), all_include_dir_from_glob.end());++ // all excluded-dir: normalize by removing trailing path separators and removing consecutive duplicates+ last = NULL;+ i = flag_all_exclude_dir.begin();+ while (i != flag_all_exclude_dir.end())+ {+ while ((i->size() > 2 || (i->size() == 2 && i->front() != '.')) && i->back() == '/')+ i->pop_back();+ if (last != NULL && i->compare(*last) == 0)+ i = flag_all_exclude_dir.erase(i);+ else+ last = &*i++;+ }++ // all included-dir: normalize by removing trailing path separators and removing consecutive duplicates+ last = NULL;+ i = flag_all_include_dir.begin();+ while (i != flag_all_include_dir.end())+ {+ while ((i->size() > 2 || (i->size() == 2 && i->front() != '.')) && i->back() == '/')+ i->pop_back();+ if (last != NULL && i->compare(*last) == 0)+ i = flag_all_include_dir.erase(i);+ else+ last = &*i++;+ }++ // if an include (sub)dir glob starts with a dot, then enable --hidden if (!flag_hidden) { for (const auto& dir : flag_all_include_dir) {- if (dir.front() == '.' || dir.find(PATHSEPSTR ".") != std::string::npos)+ if ((dir.front() == '.' && dir.compare(0, 2, "./") != 0) || dir.find("/.") != std::string::npos) { flag_hidden = true; break;@@ -7999,6 +8054,10 @@ void ugrep() } } + // if --include-dir is nonempty, and -d unspecified, then we should recurse into directories+ if (!flag_all_include_dir.empty() && flag_directories_action == Action::UNSP)+ flag_directories_action = Action::RECURSE;+ #ifdef HAVE_LIBZ #ifdef WITH_DECOMPRESSION_THREAD // -z with -M or -O/-t/-g/--include: add globs to also search archive contents for matching files@@ -8089,16 +8148,6 @@ void ugrep() #endif #endif - // all excluded-dirs: normalize by removing trailing path separators- for (auto& i : flag_all_exclude_dir)- while (i.size() > 1 && i.back() == '/')- i.pop_back();-- // all included-dirs: normalize by removing trailing path separators- for (auto& i : flag_all_include_dir)- while (i.size() > 1 && i.back() == '/')- i.pop_back();- // keep track which globs are case-insensitive when --glob-ignore-case is specified, i.e. not --ignore-file globs if (flag_glob_ignore_case) {@@ -8557,11 +8606,11 @@ void ugrep() // reflex::Matcher options std::string matcher_options; - // -Y: literally match closing ) like GNU grep when not paired with ( unless -P or -F+ // grep compatibility mode: literally match closing ) like GNU grep when not paired with ( unless -P or -F if (flag_grep && !flag_perl_regexp && !flag_fixed_strings) convert_flags |= reflex::convert_flag::closing; - // --empty: permit empty pattern matches+ // -Y or --empty: permit empty pattern matches if (flag_empty) matcher_options.push_back('N'); @@ -9154,14 +9203,15 @@ Grep::Type Grep::select(size_t level, const char *pathname, const char *basename bool ok = false; for (const auto& glob : flag_all_include_dir) {- bool ignore_case = &glob < &flag_all_include_dir.front() + flag_include_iglob_size;+ bool ignore_case = &glob < &flag_all_include_dir.front() + flag_include_iglob_dir_size; if (glob.front() == '!') { if (ok && glob_match(pathname, basename, glob.c_str() + 1, ignore_case)) ok = false; }- else if (!ok && glob_match(pathname, basename, glob.c_str(), ignore_case))+ else if (!ok && glob_match(pathname, basename, glob.c_str(), ignore_case, true)) {+ // the pathname matches the leading --include-dir glob, OK to descent into directory ok = true; } }@@ -9179,7 +9229,27 @@ Grep::Type Grep::select(size_t level, const char *pathname, const char *basename if (flag_min_depth > 0 && level <= flag_min_depth) return Type::SKIP; - bool ok = true;+ bool ok;++ if (!flag_all_include_dir.empty())+ {+ // only include paths to the file to search that match an --include-dir path glob+ ok = false;+ for (const auto& glob : flag_all_include_dir)+ {+ bool ignore_case = &glob < &flag_all_include_dir.front() + flag_include_iglob_dir_size;+ if (glob.front() != '!' && glob_match(pathname, basename, glob.c_str(), ignore_case, false, true))+ {+ // pathname matches the --include-dir path glob+ ok = true;+ break;+ }+ }+ if (!ok)+ return Type::SKIP;+ }++ ok = true; if (!flag_all_exclude.empty()) {@@ -9287,10 +9357,10 @@ Grep::Type Grep::select(size_t level, const char *pathname, const char *basename // is it a symlink? If dir entry unknown and following then set to symlink = true to call stat() below bool symlink = type != DIRENT_TYPE_UNKNOWN ? type == DIRENT_TYPE_LNK : follow ? true : S_ISLNK(buf.st_mode); // if we got a symlink, use stat() to check if pathname is a directory or a regular file, we also stat when following and when sorting by stat info such as modification time- if (( ( (type != DIRENT_TYPE_UNKNOWN && type != DIRENT_TYPE_LNK) || /* type is known and not symlink */- (!follow && !symlink) /* or not following and not symlink */+ if (( ( (type != DIRENT_TYPE_UNKNOWN && type != DIRENT_TYPE_LNK) || // type is known and not symlink+ (!follow && !symlink) // or not following and not symlink ) &&- (flag_sort_key == Sort::NA || flag_sort_key == Sort::NAME) /* and we're not sorting or by name */+ (flag_sort_key == Sort::NA || flag_sort_key == Sort::NAME) // and we're not sorting or by name ) || stat(pathname, &buf) == 0) {@@ -9347,7 +9417,7 @@ Grep::Type Grep::select(size_t level, const char *pathname, const char *basename if (!flag_all_include_dir.empty()) {- // include directories whose pathname matches any one of the --include-dir globs unless negated with !+ // include directories whose name matches any one of the --include-dir globs unless negated with ! bool ok = false; for (const auto& glob : flag_all_include_dir) {@@ -9357,8 +9427,9 @@ Grep::Type Grep::select(size_t level, const char *pathname, const char *basename if (ok && glob_match(pathname, basename, glob.c_str() + 1, ignore_case)) ok = false; }- else if (!ok && glob_match(pathname, basename, glob.c_str(), ignore_case))+ else if (!ok && glob_match(pathname, basename, glob.c_str(), ignore_case, true)) {+ // the pathname matches the leading --include-dir glob, OK to descent into directory ok = true; } }@@ -9385,7 +9456,27 @@ Grep::Type Grep::select(size_t level, const char *pathname, const char *basename if (flag_min_depth > 0 && level <= flag_min_depth) return Type::SKIP; - bool ok = true;+ bool ok;++ if (!flag_all_include_dir.empty())+ {+ // only include paths to the file to search that match an --include-dir path glob+ ok = false;+ for (const auto& glob : flag_all_include_dir)+ {+ bool ignore_case = &glob < &flag_all_include_dir.front() + flag_include_iglob_dir_size;+ if (glob.front() != '!' && glob_match(pathname, basename, glob.c_str(), ignore_case, false, true))+ {+ // pathname matches the --include-dir path glob+ ok = true;+ break;+ }+ }+ if (!ok)+ return Type::SKIP;+ }++ ok = true; if (!flag_all_exclude.empty()) {@@ -10318,8 +10409,6 @@ void Grep::search(const char *pathname, uint16_t cost) try {- size_t matches = 0;- // --files: reset the matching[] bitmask used in cnf_matching() for each matcher in matchers if (flag_files && matchers != NULL) {@@ -10349,10 +10438,7 @@ void Grep::search(const char *pathname, uint16_t cost) // --format with open/close or --max-files: we must out.acquire() lock before Stats::found_part() bool acquire = (flag_format != NULL && (flag_format_open != NULL || flag_format_close != NULL)) || (!flag_quiet && flag_max_files > 0); - bool heading = flag_with_filename;- size_t lineno = 0;-- // skip line number counting, only count matching lines+ // skip accurate line number counting, only count matching lines if (!flag_multiline && flag_max_line == 0 && flag_stats == NULL) matcher->lineno_skip(true); @@ -10443,15 +10529,15 @@ void Grep::search(const char *pathname, uint16_t cost) // --format with open/close or --max-files: we must out.acquire() lock before Stats::found_part() bool acquire = (flag_format != NULL && (flag_format_open != NULL || flag_format_close != NULL)) || flag_max_files > 0;- bool heading = flag_with_filename; + // not --match to match all or not -v if (!flag_match || !flag_invert_match) { if (flag_ungroup || flag_only_matching) { // -co or -cu: count the number of patterns matched - // skip line number counting, only count matching lines+ // skip accurate line number counting, only count matching lines if (!flag_multiline && flag_max_line == 0 && flag_stats == NULL) matcher->lineno_skip(true); @@ -10480,9 +10566,9 @@ void Grep::search(const char *pathname, uint16_t cost) } else if (!flag_multiline && !flag_invert_match && flag_max_line == 0 && matchers == NULL && flag_stats == NULL) {- // -c without -o/-u/-v/-K/-%: count the number of matching lines DO NOT KEEP LINE COUNT!+ // -c without -o/-u/-v/-K/-%: count the number of matching lines - // skip line number counting, only count matching lines+ // skip accurate line number counting, only count matching lines matcher->lineno_skip(true); while (matcher->find())@@ -10502,8 +10588,6 @@ void Grep::search(const char *pathname, uint16_t cost) { // -c without -o/-u: count the number of matching lines - size_t lineno = 0;- while (matcher->find()) { size_t current_lineno = matcher->lineno();@@ -10607,21 +10691,13 @@ void Grep::search(const char *pathname, uint16_t cost) // --format with open/close: we must out.acquire() lock before Stats::found_part() bool acquire = flag_format_open != NULL || flag_format_close != NULL;- bool heading = flag_with_filename; if (flag_invert_match) {- // FormatInvertMatchHandler requires lineno to be set precisely, i.e. after skipping --range lines- size_t lineno = flag_min_line > 0 ? flag_min_line - 1 : 0;- bool binfile = false; // unused- bool hex = false; // unused- bool binary = false; // unused- bool stop = false;- // construct event handler functor with captured *this and some of the locals- FormatInvertMatchGrepHandler invert_match_handler(*this, pathname, lineno, heading, binfile, hex, binary, matches, stop);+ FormatInvertMatchGrepHandler invert_match_handler(*this, pathname); - // register the event handler to output non-matching lines+ // register the event handler to output non-matching lines, matcher resets to NULL when a new input source is assigned matcher->set_handler(&invert_match_handler); // to get the context from the invert_match handler explicitly@@ -10678,8 +10754,8 @@ void Grep::search(const char *pathname, uint16_t cost) } else {- size_t lineno = 0;- size_t matching = 0; // format match counter per match, differs from matches counter per line+ // counter for format field %m matches+ size_t format_matching = 0; while (matcher->find()) {@@ -10705,7 +10781,10 @@ void Grep::search(const char *pathname, uint16_t cost) // --min-count: require at least min-count matches if (flag_min_count > 0 && matches < flag_min_count)+ {+ lineno = current_lineno; continue;+ } // output --format-open=FORMAT for the first (or --min-count) match found if (matches == flag_min_count + (flag_min_count == 0))@@ -10747,7 +10826,7 @@ void Grep::search(const char *pathname, uint16_t cost) // output --format=FORMAT // if the match does not span more than one line, then skip to end of the line- if (!out.format(flag_format, pathname, partname, matches, &matching, matcher, heading, matches > 1 || current_lineno == lineno, matches > 1))+ if (!out.format(flag_format, pathname, partname, matches, &format_matching, matcher, heading, matches > 1 || current_lineno == lineno, matches > 1)) if (!flag_multiline && !matcher->at_bol()) if (!matcher->skip('\n')) break;@@ -10779,15 +10858,11 @@ void Grep::search(const char *pathname, uint16_t cost) { // option -o without -ABC - size_t lineno = 0;- size_t matching = 0;- bool heading = flag_with_filename;- bool binfile = !flag_text && !flag_hex && !flag_with_hex && !flag_binary_without_match && init_is_binary();- bool hex = false;- bool binary = false;+ // counter for replace field %m matches+ size_t replace_matching = 0; bool nl = false; - // skip line number counting, only count matching lines+ // skip accurate line number counting, only count matching lines if (!flag_line_number && !flag_multiline && flag_max_line == 0 && flag_stats == NULL) matcher->lineno_skip(true); @@ -10796,7 +10871,7 @@ void Grep::search(const char *pathname, uint16_t cost) const char *begin = matcher->begin(); size_t size = matcher->size(); - binary = flag_hex || (flag_with_hex && is_binary(begin, size));+ binary = check_binary(begin, size); if (hex && !binary) {@@ -10808,14 +10883,14 @@ void Grep::search(const char *pathname, uint16_t cost) nl = false; } - size_t current_lineno = matcher->lineno();- if (nl) { out.nl(); nl = false; } + size_t current_lineno = matcher->lineno();+ // --range: max line exceeded? if (flag_max_line > 0 && current_lineno > flag_max_line) break;@@ -10836,14 +10911,13 @@ void Grep::search(const char *pathname, uint16_t cost) if (lineno != current_lineno || flag_ungroup) {+ lineno = current_lineno;+ ++matches; // --min-count: require at least min-count matches if (flag_min_count > 0 && matches < flag_min_count)- {- lineno = current_lineno; continue;- } // --max-count: max number of matches reached? if (flag_max_count > 0 && matches > flag_max_count)@@ -10878,8 +10952,6 @@ void Grep::search(const char *pathname, uint16_t cost) if (!flag_no_header) out.header(pathname, partname, heading, current_lineno, matcher, matcher->first(), flag_separator, binary); - lineno = current_lineno;- // --min-count: require at least min-count matches if (flag_min_count > 0 && matches < flag_min_count) continue;@@ -10917,7 +10989,7 @@ void Grep::search(const char *pathname, uint16_t cost) { // output --replace=FORMAT out.str(match_ms);- out.format(flag_replace, pathname, partname, matches, &matching, matcher, heading, matches > 1, matches > 1);+ out.format(flag_replace, pathname, partname, matches, &replace_matching, matcher, heading, matches > 1, matches > 1); out.str(match_off); } else@@ -10979,22 +11051,16 @@ void Grep::search(const char *pathname, uint16_t cost) size_t count = 0; size_t width = flag_before_context + flag_after_context;- size_t lineno = 0;- bool heading = flag_with_filename;- bool binfile = !flag_text && !flag_hex && !flag_with_hex && !flag_binary_without_match && init_is_binary();- bool hex = false;- bool binary = false; bool nl = false;- bool stop = false; - // skip line number counting, only count matching lines+ // skip accurate line number counting, only count matching lines if (!flag_line_number && !flag_multiline && flag_max_line == 0 && flag_stats == NULL) matcher->lineno_skip(true); - // construct event handler functor with captured *this and some of the locals- GrepHandler handler(*this, pathname, lineno, heading, binfile, hex, binary, matches, stop);+ // construct event handler functor with captured *this and some of the locals to save the restline when the buffer shifts+ OnlyMatchingContextGrepHandler handler(*this, pathname); - // register the event handler to update restline on buffer shift+ // register the event handler to update restline on buffer shift, matcher resets to NULL when a new input source is assigned matcher->set_handler(&handler); // the rest of the matching line@@ -11007,7 +11073,7 @@ void Grep::search(const char *pathname, uint16_t cost) const char *begin = matcher->begin(); size_t size = matcher->size(); - binary = flag_hex || (flag_with_hex && is_binary(begin, size));+ binary = check_binary(begin, size); if (hex && !binary) {@@ -11019,8 +11085,6 @@ void Grep::search(const char *pathname, uint16_t cost) nl = false; } - size_t current_lineno = matcher->lineno();- if (nl) { if (restline_size > 0)@@ -11060,6 +11124,8 @@ void Grep::search(const char *pathname, uint16_t cost) nl = false; } + size_t current_lineno = matcher->lineno();+ // --range: max line exceeded? if (flag_max_line > 0 && current_lineno > flag_max_line) break;@@ -11080,14 +11146,13 @@ void Grep::search(const char *pathname, uint16_t cost) if (lineno != current_lineno || flag_ungroup) {+ lineno = current_lineno;+ ++matches; // --min-count: require at least min-count matches if (flag_min_count > 0 && matches < flag_min_count)- {- lineno = current_lineno; continue;- } // --max-count: max number of matches reached? if (flag_max_count > 0 && matches > flag_max_count)@@ -11119,10 +11184,11 @@ void Grep::search(const char *pathname, uint16_t cost) goto done_search; } - if (!flag_no_header)- out.header(pathname, partname, heading, current_lineno, matcher, matcher->first(), flag_separator, binary);+ size_t first = matcher->first();+ size_t last = matcher->last(); - lineno = current_lineno;+ if (!flag_no_header)+ out.header(pathname, partname, heading, current_lineno, matcher, first, flag_separator, binary); // --min-count: require at least min-count matches if (flag_min_count > 0 && matches < flag_min_count)@@ -11134,8 +11200,8 @@ void Grep::search(const char *pathname, uint16_t cost) { if (flag_hex || flag_with_hex) {- out.dump.next(matcher->first());- out.dump.hex(Output::Dump::HEX_MATCH, matcher->first(), begin, size);+ out.dump.next(first);+ out.dump.hex(Output::Dump::HEX_MATCH, first, begin, size); } else {@@ -11161,11 +11227,11 @@ void Grep::search(const char *pathname, uint16_t cost) { if (restline_size > 0) {- size_t first = matcher->first(); if (first - restline_last < restline_size) restline_size = first - restline_last; size_t after = utf8nlen(restline_data, restline_size);+ if (after > width) after = width; @@ -11177,6 +11243,7 @@ void Grep::search(const char *pathname, uint16_t cost) out.utf8strn(restline_data, restline_size, after); out.str(color_off); }+ restline_size = 0; } @@ -11272,7 +11339,7 @@ void Grep::search(const char *pathname, uint16_t cost) // output header and repeat to output a multiline match after previous newline if (!flag_no_header)- out.header(pathname, partname, heading, ++lineno, matcher, matcher->first() + (to - begin), flag_separator_bar, false);+ out.header(pathname, partname, heading, ++lineno, matcher, first + (to - begin), flag_separator_bar, false); begin = to + 1; size = rest - 1;@@ -11286,7 +11353,7 @@ void Grep::search(const char *pathname, uint16_t cost) restline_data = end; restline_size = eol - end;- restline_last = matcher->last();+ restline_last = last; } nl = true;@@ -11339,23 +11406,18 @@ void Grep::search(const char *pathname, uint16_t cost) { // options -ABC, -y, -v are not specified, --hexdump context is supported (including with options -ABC) - size_t lineno = 0;- size_t matching = 0;- bool heading = flag_with_filename;- bool binfile = !flag_text && !flag_hex && !flag_with_hex && !flag_binary_without_match && init_is_binary();- bool hex = false;- bool binary = false;- bool stop = false;+ // counter for replace field %m matches+ size_t replace_matching = 0; bool colorize = flag_color != NULL || flag_replace != NULL || flag_tag != NULL; - // skip line number counting, only count matching lines+ // skip accurate line number counting, only count matching lines if (!flag_line_number && !flag_multiline && flag_max_line == 0 && flag_stats == NULL) matcher->lineno_skip(true); - // construct event handler functor with captured *this and some of the locals- GrepHandler handler(*this, pathname, lineno, heading, binfile, hex, binary, matches, stop);+ // construct event handler functor with captured *this and some of the locals to output or save restline when the buffer shifts+ GrepHandler handler(*this, pathname); - // register the event handler to update restline on buffer shift+ // register the event handler to update restline on buffer shift, matcher resets to NULL when a new input source is assigned matcher->set_handler(&handler); // --hexdump: keep -B NUM+1 before context times hex column bytes in the buffer when shifting@@ -11373,43 +11435,7 @@ void Grep::search(const char *pathname, uint16_t cost) if (lineno != current_lineno || flag_ungroup) {- if (restline_data != NULL)- {- if (binary)- {- if (flag_hex_after > 0)- {- size_t right = flag_hex_after * flag_hex_columns - ((restline_last - 1) % flag_hex_columns) - 1;- if (restline_last + right > matcher->first())- right = matcher->first() - restline_last;- if (right < restline_size)- restline_size = right;- }-- out.dump.hex(Output::Dump::HEX_LINE, restline_last, restline_data, restline_size);-- if (lineno + 1 < current_lineno)- out.dump.done();- }- else- {- bool lf_only = false;- if (restline_size > 0)- {- lf_only = restline_data[restline_size - 1] == '\n';- restline_size -= lf_only;- if (restline_size > 0)- {- out.str(color_sl);- out.str(restline_data, restline_size);- out.str(color_off);- }- }- out.nl(lf_only);- }-- restline_data = NULL;- }+ output_restline(false, flag_hex_after > 0); // --range: max line exceeded? if (flag_max_line > 0 && current_lineno > flag_max_line)@@ -11446,8 +11472,9 @@ void Grep::search(const char *pathname, uint16_t cost) goto exit_search; } - binary = flag_hex || (flag_with_hex && is_binary(bol, eol - bol));+ binary = check_binary(bol, eol - bol); + // --hexdump with ABC context: if binary then show hex context if (binary && (flag_hex_after > 0 || flag_hex_before > 0)) { const char *aft = matcher->aft((flag_hex_after + flag_hex_before) * flag_hex_columns);@@ -11478,6 +11505,7 @@ void Grep::search(const char *pathname, uint16_t cost) size_t border = matcher->border(); size_t first = matcher->first();+ size_t last = matcher->last(); const char *begin = matcher->begin(); const char *end = matcher->end(); size_t size = matcher->size();@@ -11494,6 +11522,7 @@ void Grep::search(const char *pathname, uint16_t cost) if (binary) {+ // --hexdump with ABC context: if binary then show hex context if (flag_hex_before > 0) { const char *bef = matcher->bef(flag_hex_before * flag_hex_columns);@@ -11523,21 +11552,22 @@ void Grep::search(const char *pathname, uint16_t cost) if (flag_ungroup) {+ // --hexdump with ABC context: if binary then show hex context if (flag_hex_after > 0) {- size_t right = flag_hex_after * flag_hex_columns - ((matcher->last() - 1) % flag_hex_columns) - 1;+ size_t right = flag_hex_after * flag_hex_columns - ((last - 1) % flag_hex_columns) - 1; if (end + right < eol) eol = end + right; } - out.dump.hex(Output::Dump::HEX_LINE, matcher->last(), end, eol - end);+ out.dump.hex(Output::Dump::HEX_LINE, last, end, eol - end); out.dump.done(); } else { restline_data = end; restline_size = eol - end;- restline_last = matcher->last();+ restline_last = last; } lineno += matcher->lines() - 1;@@ -11555,8 +11585,10 @@ void Grep::search(const char *pathname, uint16_t cost) { // output --replace=FORMAT out.str(match_ms);- out.format(flag_replace, pathname, partname, matches, &matching, matcher, heading, matches > 1, matches > 1);+ out.format(flag_replace, pathname, partname, matches, &replace_matching, matcher, heading, matches > 1, matches > 1); out.str(match_off);++ lineno += matcher->lines() - 1; } else {@@ -11614,7 +11646,7 @@ void Grep::search(const char *pathname, uint16_t cost) { restline_data = end; restline_size = eol - end;- restline_last = matcher->last();+ restline_last = last; } } @@ -11649,6 +11681,7 @@ void Grep::search(const char *pathname, uint16_t cost) if (restline_last + restline_size > first) restline_size = first - restline_last; + // --hexdump with ABC context: if binary then show hex context if (flag_hex_after > 0) { size_t right = flag_hex_after * flag_hex_columns - ((restline_last - 1) % flag_hex_columns) - 1;@@ -11670,6 +11703,7 @@ void Grep::search(const char *pathname, uint16_t cost) if (restline_data != NULL) {+ // --hexdump with ABC context: if binary then show hex context if (flag_hex_before > 0) { size_t left = flag_hex_before * flag_hex_columns + (first % flag_hex_columns) - flag_hex_columns;@@ -11678,8 +11712,8 @@ void Grep::search(const char *pathname, uint16_t cost) if (!flag_no_header) out.header(pathname, partname, heading, current_lineno, matcher, first, flag_separator_bar, binary); - restline_data = restline_data + restline_size - left;- restline_last = restline_last + restline_size - left;+ restline_data += restline_size - left;+ restline_last += restline_size - left; restline_size = left; } }@@ -11694,15 +11728,18 @@ void Grep::search(const char *pathname, uint16_t cost) } else {- out.str(color_sl);- out.str(restline_data, first - restline_last);- out.str(color_off);+ if (restline_data != NULL)+ {+ out.str(color_sl);+ out.str(restline_data, first - restline_last);+ out.str(color_off);+ } if (flag_replace != NULL) { // output --replace=FORMAT out.str(match_ms);- out.format(flag_replace, pathname, partname, matches, &matching, matcher, heading, matches > 1, matches > 1);+ out.format(flag_replace, pathname, partname, matches, &replace_matching, matcher, heading, matches > 1, matches > 1); out.str(match_off); } else@@ -11743,6 +11780,7 @@ void Grep::search(const char *pathname, uint16_t cost) if (lines == 1) {+ // --hexdump with ABC context: if binary then show hex context if (binary && (flag_hex_after > 0 || flag_hex_before > 0)) { const char *aft = matcher->aft((flag_hex_after + flag_hex_before) * flag_hex_columns);@@ -11759,8 +11797,9 @@ void Grep::search(const char *pathname, uint16_t cost) } else {- binary = flag_hex || (flag_with_hex && is_binary(end, eol - end));+ binary = check_binary(end, eol - end); + // --hexdump with ABC context: if binary then show hex context if (binary && (flag_hex_after > 0 || flag_hex_before > 0)) { const char *aft = matcher->aft((flag_hex_after + flag_hex_before) * flag_hex_columns);@@ -11785,14 +11824,15 @@ void Grep::search(const char *pathname, uint16_t cost) { if (binary) {+ // --hexdump with ABC context: if binary then show hex context if (flag_hex_after > 0) {- size_t right = flag_hex_after * flag_hex_columns - ((matcher->last() - 1) % flag_hex_columns) - 1;+ size_t right = flag_hex_after * flag_hex_columns - ((last - 1) % flag_hex_columns) - 1; if (end + right < eol) eol = end + right; } - out.dump.hex(Output::Dump::HEX_LINE, matcher->last(), end, eol - end);+ out.dump.hex(Output::Dump::HEX_LINE, last, end, eol - end); out.dump.done(); } else@@ -11837,38 +11877,7 @@ void Grep::search(const char *pathname, uint16_t cost) if (flag_min_count > 0 && matches < flag_min_count) goto exit_search; - if (restline_data != NULL)- {- if (binary)- {- if (flag_hex_after > 0)- {- size_t right = flag_hex_after * flag_hex_columns - ((restline_last - 1) % flag_hex_columns) - 1;- if (right < restline_size)- restline_size = right;- }-- out.dump.hex(Output::Dump::HEX_LINE, restline_last, restline_data, restline_size);- }- else- {- bool lf_only = false;- if (restline_size > 0)- {- lf_only = restline_data[restline_size - 1] == '\n';- restline_size -= lf_only;- if (restline_size > 0)- {- out.str(color_sl);- out.str(restline_data, restline_size);- out.str(color_off);- }- }- out.nl(lf_only);- }-- restline_data = NULL;- }+ output_restline(false, flag_hex_after > 0); if (binary) out.dump.done();@@ -11877,18 +11886,10 @@ void Grep::search(const char *pathname, uint16_t cost) { // option -v without -ABC, -y - // InvertMatchHandler requires lineno to be set precisely, i.e. after skipping --range lines- size_t lineno = flag_min_line > 0 ? flag_min_line - 1 : 0;- bool heading = flag_with_filename;- bool binfile = !flag_text && !flag_hex && !flag_with_hex && !flag_binary_without_match && init_is_binary();- bool hex = false;- bool binary = false;- bool stop = false;+ // construct event handler functor with captured *this and some of the locals to output or save restline when the buffer shifts+ InvertMatchGrepHandler invert_match_handler(*this, pathname); - // construct event handler functor with captured *this and some of the locals- InvertMatchGrepHandler invert_match_handler(*this, pathname, lineno, heading, binfile, hex, binary, matches, stop);-- // register the event handler to output non-matching lines+ // register the event handler to output non-matching lines, matcher resets to NULL when a new input source is assigned matcher->set_handler(&invert_match_handler); // to get the context from the invert_match handler explicitly@@ -11964,14 +11965,8 @@ void Grep::search(const char *pathname, uint16_t cost) { // option -y - // AnyLineGrepHandler requires lineno to be set precisely, i.e. after skipping --range lines- size_t lineno = flag_min_line > 0 ? flag_min_line - 1 : 0;- size_t matching = 0;- bool heading = flag_with_filename;- bool binfile = !flag_text && !flag_hex && !flag_with_hex && !flag_binary_without_match && init_is_binary();- bool hex = false;- bool binary = false;- bool stop = false;+ // counter for replace field %m matches+ size_t replace_matching = 0; bool colorize = flag_color != NULL || flag_replace != NULL || flag_tag != NULL; // to output the rest of the matching line@@ -11979,10 +11974,10 @@ void Grep::search(const char *pathname, uint16_t cost) restline_size = 0; restline_last = 0; - // construct event handler functor with captured *this and some of the locals- AnyLineGrepHandler any_line_handler(*this, pathname, lineno, heading, binfile, hex, binary, matches, stop);+ // construct event handler functor with captured *this and some of the locals to output or save restline when the buffer shifts+ AnyLineGrepHandler any_line_handler(*this, pathname); - // register the event handler functor to output non-matching lines+ // register the event handler functor to output non-matching lines, matcher resets to NULL when a new input source is assigned matcher->set_handler(&any_line_handler); // to output colors with or without -v@@ -12001,31 +11996,7 @@ void Grep::search(const char *pathname, uint16_t cost) if (lineno != current_lineno) /* || flag_ungroup) // logically OK but dead code because -y */ {- if (restline_data != NULL)- {- if (binary)- {- out.dump.hex(v_hex_line, restline_last, restline_data, restline_size);- }- else- {- bool lf_only = false;- if (restline_size > 0)- {- lf_only = restline_data[restline_size - 1] == '\n';- restline_size -= lf_only;- if (restline_size > 0)- {- out.str(v_color_sl);- out.str(restline_data, restline_size);- out.str(color_off);- }- }- out.nl(lf_only);- }-- restline_data = NULL;- }+ output_restline(flag_invert_match); const char *eol = matcher->eol(true); // warning: call eol() before bol() and end() const char *bol = matcher->bol();@@ -12095,7 +12066,7 @@ void Grep::search(const char *pathname, uint16_t cost) if (out.eof) goto exit_search; - binary = flag_hex || (flag_with_hex && is_binary(bol, eol - bol));+ binary = check_binary(bol, eol - bol); if (binfile || (binary && !flag_hex && !flag_with_hex)) {@@ -12122,6 +12093,7 @@ void Grep::search(const char *pathname, uint16_t cost) size_t border = matcher->border(); size_t first = matcher->first();+ size_t last = matcher->last(); const char *begin = matcher->begin(); const char *end = matcher->end(); size_t size = matcher->size();@@ -12143,14 +12115,14 @@ void Grep::search(const char *pathname, uint16_t cost) if (false) /* flag_ungroup) // logically OK but dead code because -y */ {- out.dump.hex(v_hex_line, matcher->last(), end, eol - end);+ out.dump.hex(v_hex_line, last, end, eol - end); out.dump.done(); } else { restline_data = end; restline_size = eol - end;- restline_last = matcher->last();+ restline_last = last; } lineno += matcher->lines() - 1;@@ -12168,7 +12140,7 @@ void Grep::search(const char *pathname, uint16_t cost) { // output --replace=FORMAT out.str(v_match_ms);- out.format(flag_replace, pathname, partname, matches, &matching, matcher, heading, matches > 1, matches > 1);+ out.format(flag_replace, pathname, partname, matches, &replace_matching, matcher, heading, matches > 1, matches > 1); out.str(match_off); lineno += matcher->lines() - 1;@@ -12229,7 +12201,7 @@ void Grep::search(const char *pathname, uint16_t cost) { restline_data = end; restline_size = eol - end;- restline_last = matcher->last();+ restline_last = last; } } @@ -12259,20 +12231,25 @@ void Grep::search(const char *pathname, uint16_t cost) if (binary) {- out.dump.hex(v_hex_line, restline_last, restline_data, first - restline_last);+ if (restline_data != NULL)+ out.dump.hex(v_hex_line, restline_last, restline_data, first - restline_last);+ out.dump.hex(v_hex_match, first, begin, size); } else {- out.str(v_color_sl);- out.str(restline_data, first - restline_last);- out.str(color_off);+ if (restline_data != NULL)+ {+ out.str(v_color_sl);+ out.str(restline_data, first - restline_last);+ out.str(color_off);+ } if (flag_replace != NULL) { // output --replace=FORMAT out.str(v_match_ms);- out.format(flag_replace, pathname, partname, matches, &matching, matcher, heading, matches > 1, matches > 1);+ out.format(flag_replace, pathname, partname, matches, &replace_matching, matcher, heading, matches > 1, matches > 1); out.str(match_off); } else@@ -12308,18 +12285,18 @@ void Grep::search(const char *pathname, uint16_t cost) } } + const char *eol = matcher->eol(true); // warning: call eol() before end()+ const char *end = matcher->end();+ if (lines == 1) {- restline_data += last - restline_last;- restline_size -= last - restline_last;+ restline_data = end;+ restline_size = eol - end; restline_last = last; } else {- const char *eol = matcher->eol(true); // warning: call eol() before end()- const char *end = matcher->end();-- binary = flag_hex || (flag_with_hex && is_binary(end, eol - end));+ binary = check_binary(end, eol - end); if (hex && !binary) out.dump.done();@@ -12335,7 +12312,7 @@ void Grep::search(const char *pathname, uint16_t cost) { if (binary) {- out.dump.hex(v_hex_line, matcher->last(), end, eol - end);+ out.dump.hex(v_hex_line, last, end, eol - end); out.dump.done(); } else@@ -12380,31 +12357,7 @@ void Grep::search(const char *pathname, uint16_t cost) if (flag_min_count > 0 && matches < flag_min_count) matches = 0; - if (restline_data != NULL)- {- if (binary)- {- out.dump.hex(v_hex_line, restline_last, restline_data, restline_size);- }- else- {- bool lf_only = false;- if (restline_size > 0)- {- lf_only = restline_data[restline_size - 1] == '\n';- restline_size -= lf_only;- if (restline_size > 0)- {- out.str(v_color_sl);- out.str(restline_data, restline_size);- out.str(color_off);- }- }- out.nl(lf_only);- }-- restline_data = NULL;- }+ output_restline(flag_invert_match); // get the remaining context after the last match to output context = matcher->after();@@ -12427,14 +12380,8 @@ void Grep::search(const char *pathname, uint16_t cost) { // options -ABC without -v - // ContextGrepHandler requires lineno to be set precisely, i.e. after skipping --range lines- size_t lineno = flag_min_line > 0 ? flag_min_line - 1 : 0;- size_t matching = 0;- bool heading = flag_with_filename;- bool binfile = !flag_text && !flag_hex && !flag_with_hex && !flag_binary_without_match && init_is_binary();- bool hex = false;- bool binary = false;- bool stop = false;+ // counter for replace field %m matches+ size_t replace_matching = 0; bool colorize = flag_color != NULL || flag_replace != NULL || flag_tag != NULL; // to output the rest of the matching line@@ -12442,10 +12389,10 @@ void Grep::search(const char *pathname, uint16_t cost) restline_size = 0; restline_last = 0; - // construct event handler functor with captured *this and some of the locals- ContextGrepHandler context_handler(*this, pathname, lineno, heading, binfile, hex, binary, matches, stop);+ // construct event handler functor with captured *this and some of the locals to output or save restline when the buffer shifts+ ContextGrepHandler context_handler(*this, pathname); - // register the event handler functor to output non-matching lines+ // register the event handler functor to output non-matching lines, matcher resets to NULL when a new input source is assigned matcher->set_handler(&context_handler); // to get the context from the context handler explicitly@@ -12457,31 +12404,7 @@ void Grep::search(const char *pathname, uint16_t cost) if (lineno != current_lineno || flag_ungroup) {- if (restline_data != NULL)- {- if (binary)- {- out.dump.hex(Output::Dump::HEX_LINE, restline_last, restline_data, restline_size);- }- else- {- bool lf_only = false;- if (restline_size > 0)- {- lf_only = restline_data[restline_size - 1] == '\n';- restline_size -= lf_only;- if (restline_size > 0)- {- out.str(color_sl);- out.str(restline_data, restline_size);- out.str(color_off);- }- }- out.nl(lf_only);- }-- restline_data = NULL;- }+ output_restline(); const char *eol = matcher->eol(true); // warning: call eol() before bol() and end() const char *bol = matcher->bol();@@ -12538,7 +12461,7 @@ void Grep::search(const char *pathname, uint16_t cost) context_handler.output_before_context(); - binary = flag_hex || (flag_with_hex && is_binary(bol, eol - bol));+ binary = check_binary(bol, eol - bol); if (binfile || (binary && !flag_hex && !flag_with_hex)) {@@ -12560,6 +12483,7 @@ void Grep::search(const char *pathname, uint16_t cost) size_t border = matcher->border(); size_t first = matcher->first();+ size_t last = matcher->last(); const char *begin = matcher->begin(); const char *end = matcher->end(); size_t size = matcher->size();@@ -12581,14 +12505,14 @@ void Grep::search(const char *pathname, uint16_t cost) if (flag_ungroup) {- out.dump.hex(Output::Dump::HEX_LINE, matcher->last(), end, eol - end);+ out.dump.hex(Output::Dump::HEX_LINE, last, end, eol - end); out.dump.done(); } else { restline_data = end; restline_size = eol - end;- restline_last = matcher->last();+ restline_last = last; } lineno += matcher->lines() - 1;@@ -12606,7 +12530,7 @@ void Grep::search(const char *pathname, uint16_t cost) { // output --replace=FORMAT out.str(match_ms);- out.format(flag_replace, pathname, partname, matches, &matching, matcher, heading, matches > 1, matches > 1);+ out.format(flag_replace, pathname, partname, matches, &replace_matching, matcher, heading, matches > 1, matches > 1); out.str(match_off); lineno += matcher->lines() - 1;@@ -12667,7 +12591,7 @@ void Grep::search(const char *pathname, uint16_t cost) { restline_data = end; restline_size = eol - end;- restline_last = matcher->last();+ restline_last = last; } } @@ -12712,7 +12636,7 @@ void Grep::search(const char *pathname, uint16_t cost) { // output --replace=FORMAT out.str(match_ms);- out.format(flag_replace, pathname, partname, matches, &matching, matcher, heading, matches > 1, matches > 1);+ out.format(flag_replace, pathname, partname, matches, &replace_matching, matcher, heading, matches > 1, matches > 1); out.str(match_off); } else@@ -12748,18 +12672,18 @@ void Grep::search(const char *pathname, uint16_t cost) } } + const char *eol = matcher->eol(true); // warning: call eol() before end()+ const char *end = matcher->end();+ if (lines == 1) {- restline_data += last - restline_last;- restline_size -= last - restline_last;+ restline_data = end;+ restline_size = eol - end; restline_last = last; } else {- const char *eol = matcher->eol(true); // warning: call eol() before end()- const char *end = matcher->end();-- binary = flag_hex || (flag_with_hex && is_binary(end, eol - end));+ binary = check_binary(end, eol - end); if (hex && !binary) out.dump.done();@@ -12775,7 +12699,7 @@ void Grep::search(const char *pathname, uint16_t cost) { if (binary) {- out.dump.hex(Output::Dump::HEX_LINE, matcher->last(), end, eol - end);+ out.dump.hex(Output::Dump::HEX_LINE, last, end, eol - end); out.dump.done(); } else@@ -12818,31 +12742,7 @@ void Grep::search(const char *pathname, uint16_t cost) context_handler.set_after_lineno(lineno + 1); } - if (restline_data != NULL)- {- if (binary)- {- out.dump.hex(Output::Dump::HEX_LINE, restline_last, restline_data, restline_size);- }- else- {- bool lf_only = false;- if (restline_size > 0)- {- lf_only = restline_data[restline_size - 1] == '\n';- restline_size -= lf_only;- if (restline_size > 0)- {- out.str(color_sl);- out.str(restline_data, restline_size);- out.str(color_off);- }- }- out.nl(lf_only);- }-- restline_data = NULL;- }+ output_restline(); // get the remaining context after the last match to output context = matcher->after();@@ -12865,17 +12765,11 @@ void Grep::search(const char *pathname, uint16_t cost) { // options -ABC with -v - // InvertContextGrepHandler requires lineno to be set precisely, i.e. after skipping --range lines- size_t lineno = flag_min_line > 0 ? flag_min_line - 1 : 0;+ size_t after = flag_after_context; size_t last_lineno = lineno; size_t next_lineno = lineno + 1;- size_t matching = 0;- size_t after = flag_after_context;- bool heading = flag_with_filename;- bool binfile = !flag_text && !flag_hex && !flag_with_hex && !flag_binary_without_match && init_is_binary();- bool hex = false;- bool binary = false;- bool stop = false;+ // counter for replace field %m matches+ size_t replace_matching = 0; bool colorize = flag_color != NULL || flag_replace != NULL || flag_tag != NULL; // to output the rest of the matching line@@ -12883,10 +12777,10 @@ void Grep::search(const char *pathname, uint16_t cost) restline_size = 0; restline_last = 0; - // construct event handler functor with captured *this and some of the locals- InvertContextGrepHandler invert_context_handler(*this, pathname, lineno, heading, binfile, hex, binary, matches, stop);+ // construct event handler functor with captured *this and some of the locals to output or save restline when the buffer shifts+ InvertContextGrepHandler invert_context_handler(*this, pathname); - // register the event handler functor to output non-matching lines+ // register the event handler functor to output non-matching lines, matcher resets to NULL when a new input source is assigned matcher->set_handler(&invert_context_handler); // to get the context from the context handler explicitly@@ -12908,31 +12802,7 @@ void Grep::search(const char *pathname, uint16_t cost) if (lineno != current_lineno) {- if (restline_data != NULL)- {- if (binary)- {- out.dump.hex(Output::Dump::HEX_CONTEXT_LINE, restline_last, restline_data, restline_size);- }- else- {- bool lf_only = false;- if (restline_size > 0)- {- lf_only = restline_data[restline_size - 1] == '\n';- restline_size -= lf_only;- if (restline_size > 0)- {- out.str(color_cx);- out.str(restline_data, restline_size);- out.str(color_off);- }- }- out.nl(lf_only);- }-- restline_data = NULL;- }+ output_restline(true); const char *eol = matcher->eol(true); // warning: call eol() before bol() and end() const char *bol = matcher->bol();@@ -12993,7 +12863,7 @@ void Grep::search(const char *pathname, uint16_t cost) if (after < flag_after_context) {- binary = flag_hex || (flag_with_hex && is_binary(bol, eol - bol));+ binary = check_binary(bol, eol - bol); if (binfile || (binary && !flag_hex && !flag_with_hex)) {@@ -13021,6 +12891,7 @@ void Grep::search(const char *pathname, uint16_t cost) size_t border = matcher->border(); size_t first = matcher->first();+ size_t last = matcher->last(); const char *begin = matcher->begin(); const char *end = matcher->end(); size_t size = matcher->size();@@ -13042,7 +12913,7 @@ void Grep::search(const char *pathname, uint16_t cost) restline_data = end; restline_size = eol - end;- restline_last = matcher->last();+ restline_last = last; } } else@@ -13058,7 +12929,7 @@ void Grep::search(const char *pathname, uint16_t cost) { // output --replace=FORMAT out.str(match_mc);- out.format(flag_replace, pathname, partname, matches, &matching, matcher, heading, matches > 1, matches > 1);+ out.format(flag_replace, pathname, partname, matches, &replace_matching, matcher, heading, matches > 1, matches > 1); out.str(match_off); } else@@ -13095,12 +12966,12 @@ void Grep::search(const char *pathname, uint16_t cost) restline_data = end; restline_size = eol - end;- restline_last = matcher->last();+ restline_last = last; } } else if (flag_before_context > 0) {- binary = flag_hex || (flag_with_hex && is_binary(bol, eol - bol));+ binary = check_binary(bol, eol - bol); if (binfile || (binary && !flag_hex && !flag_with_hex)) {@@ -13180,7 +13051,8 @@ void Grep::search(const char *pathname, uint16_t cost) if (binary) {- out.dump.hex(Output::Dump::HEX_CONTEXT_LINE, restline_last, restline_data, first - restline_last);+ if (restline_data != NULL)+ out.dump.hex(Output::Dump::HEX_CONTEXT_LINE, restline_last, restline_data, first - restline_last); out.dump.hex(Output::Dump::HEX_CONTEXT_MATCH, first, begin, size); } else@@ -13193,7 +13065,7 @@ void Grep::search(const char *pathname, uint16_t cost) { // output --replace=FORMAT out.str(match_mc);- out.format(flag_replace, pathname, partname, matches, &matching, matcher, heading, matches > 1, matches > 1);+ out.format(flag_replace, pathname, partname, matches, &replace_matching, matcher, heading, matches > 1, matches > 1); out.str(match_off); } else@@ -13240,7 +13112,7 @@ void Grep::search(const char *pathname, uint16_t cost) const char *eol = matcher->eol(true); // warning: call eol() before end() const char *end = matcher->end(); - binary = flag_hex || (flag_with_hex && is_binary(end, eol - end));+ binary = check_binary(end, eol - end); if (hex && !binary) out.dump.done();@@ -13261,31 +13133,7 @@ void Grep::search(const char *pathname, uint16_t cost) } else {- if (restline_data != NULL)- {- if (binary)- {- out.dump.hex(Output::Dump::HEX_CONTEXT_LINE, restline_last, restline_data, restline_size);- }- else- {- bool lf_only = false;- if (restline_size > 0)- {- lf_only = restline_data[restline_size - 1] == '\n';- restline_size -= lf_only;- if (restline_size > 0)- {- out.str(color_cx);- out.str(restline_data, restline_size);- out.str(color_off);- }- }- out.nl(lf_only);- }-- restline_data = NULL;- }+ output_restline(true); if (flag_before_context > 0) {@@ -13305,7 +13153,7 @@ void Grep::search(const char *pathname, uint16_t cost) const char *end = matcher->end(); - binary = flag_hex || (flag_with_hex && is_binary(end, eol - end));+ binary = check_binary(end, eol - end); if (binfile || (binary && !flag_hex && !flag_with_hex)) {@@ -13363,31 +13211,7 @@ void Grep::search(const char *pathname, uint16_t cost) lineno = next_lineno - 1; } - if (restline_data != NULL)- {- if (binary)- {- out.dump.hex(Output::Dump::HEX_CONTEXT_LINE, restline_last, restline_data, restline_size);- }- else- {- bool lf_only = false;- if (restline_size > 0)- {- lf_only = restline_data[restline_size - 1] == '\n';- restline_size -= lf_only;- if (restline_size > 0)- {- out.str(color_cx);- out.str(restline_data, restline_size);- out.str(color_off);- }- }- out.nl(lf_only);- }-- restline_data = NULL;- }+ output_restline(true); // get the remaining context after the last match to output context = matcher->after();@@ -14119,7 +13943,7 @@ void help(std::ostream& out) Specify a PATTERN to search the input. An input line is selected\n\ if it matches any of the specified patterns. This option is useful\n\ when multiple -e options are used to specify multiple patterns, or\n\- when a pattern begins with a dash (`-'), or to specify a pattern\n\+ when a pattern starts with a dash (`-'), or to specify a pattern\n\ after option -f or after the FILE arguments.\n\ --encoding=ENCODING\n\ The encoding format of the input. The default ENCODING is binary\n\@@ -14152,7 +13976,7 @@ void help(std::ostream& out) quoted to prevent shell globbing. This option may be repeated.\n\ --exclude-dir=GLOB\n\ Exclude directories whose name matches GLOB from recursive\n\- searches, same as -g ^GLOB/. GLOB can use **, *, ?, and [...] as\n\+ searches, same as -g ^GLOB/. GLOB may use **, *, ?, and [...] as\n\ wildcards and \\ to quote a wildcard or backslash character\n\ literally. When GLOB contains a `/', full pathnames are matched.\n\ Otherwise basenames are matched. Note that --exclude-dir patterns\n\@@ -14160,7 +13984,7 @@ void help(std::ostream& out) to prevent shell globbing. This option may be repeated.\n\ --exclude-from=FILE\n\ Read the globs from FILE and skip files and directories whose name\n\- matches one or more globs. A glob can use **, *, ?, and [...] as\n\+ matches one or more globs. A glob may use **, *, ?, and [...] as\n\ wildcards and \\ to quote a wildcard or backslash character\n\ literally. When a glob contains a `/', full pathnames are matched.\n\ Otherwise basenames are matched. When a glob ends with a `/',\n\@@ -14233,20 +14057,26 @@ void help(std::ostream& out) is also enabled. Context options -A, -B, -C and -y are ignored.\n\ --free-space\n\ Spacing (blanks and tabs) in regular expressions are ignored.\n\+ --from=FILE\n\+ Read additional pathnames of files to search from FILE. When FILE\n\+ is a `-', standard input is read. This option is useful with `find\n\+ ... -print | ugrep --from=- ...' to search specific files.\n\ -G, --basic-regexp\n\ Interpret patterns as basic regular expressions (BREs).\n\ -g GLOBS, --glob=GLOBS, --iglob=GLOBS\n\ Only search files whose name matches the specified comma-separated\n\- list of GLOBS, same as --include=glob for each `glob' in GLOBS.\n\+ list of GLOBS, same as --include='glob' for each `glob' in GLOBS.\n\ When a `glob' is preceded by a `!' or a `^', skip files whose name\n\ matches `glob', same as --exclude='glob'. When `glob' contains a\n\- `/', full pathnames are matched. Otherwise basenames are matched.\n\- When `glob' ends with a `/', directories are matched, same as\n\- --include-dir='glob' and --exclude-dir='glob'. A leading `/'\n\- matches the working directory. Option --iglob performs\n\- case-insensitive name matching. This option may be repeated and\n\- may be combined with options -M, -O and -t. For more details, see\n\- `ugrep --help globs' and `man ugrep' section GLOBBING for details.\n\+ `/', full pathnames are matched and option -r is enabled to recurse\n\+ into directories if -d, -r or -R is not already specified.\n\+ Otherwise basenames are matched. When `glob' ends with a `/',\n\+ directories are matched, same as --include-dir='glob' and\n\+ --exclude-dir='glob'. A leading `./' or just `/' matches the\n\+ working directory. Option --iglob=GLOBS performs case-insensitive\n\+ name matching. This option may be repeated and may be combined\n\+ with options -M, -O and -t. For more details, see `ugrep --help\n\+ globs' and `man ugrep' section GLOBBING for details.\n\ --glob-ignore-case\n\ Perform case-insensitive glob matching in general.\n\ --group-separator[=SEP]\n\@@ -14314,24 +14144,28 @@ void help(std::ostream& out) Do not ignore files, i.e. cancel --ignore-files when specified.\n\ --include=GLOB\n\ Only search files whose name matches GLOB, same as -g GLOB. GLOB\n\- can use **, *, ?, and [...] as wildcards and \\ to quote a wildcard\n\+ may use **, *, ?, and [...] as wildcards and \\ to quote a wildcard\n\ or backslash character literally. When GLOB contains a `/', full\n\- pathnames are matched. Otherwise basenames are matched. When GLOB\n\- ends with a `/', directories are included as if --include-dir is\n\- specified. Otherwise files are included. Note that --exclude\n\- patterns take priority over --include patterns. GLOB should be\n\- quoted to prevent shell globbing. This option may be repeated.\n\+ pathnames are matched and option -r is enabled to recurse into\n\+ directories if -d, -r or -R is not already specified. Otherwise\n\+ basenames are matched. When GLOB ends with a `/', directories are\n\+ included as if --include-dir is specified. Otherwise files are\n\+ included. Note that --exclude patterns take priority over\n\+ --include patterns. GLOB should be quoted to prevent shell\n\+ globbing. This option may be repeated.\n\ --include-dir=GLOB\n\ Only directories whose name matches GLOB are included in recursive\n\- searches, same as -g GLOB/. GLOB can use **, *, ?, and [...] as\n\+ searches, same as -g GLOB/. GLOB may use **, *, ?, and [...] as\n\ wildcards and \\ to quote a wildcard or backslash character\n\- literally. When GLOB contains a `/', full pathnames are matched.\n\- Otherwise basenames are matched. Note that --exclude-dir patterns\n\- take priority over --include-dir patterns. GLOB should be quoted\n\- to prevent shell globbing. This option may be repeated.\n\+ literally. When GLOB contains a `/', full pathnames are matched\n\+ and option -r is enabled to recurse into directories if -d, -r or\n\+ -R is not already specified. Otherwise basenames are matched.\n\+ Note that --exclude-dir patterns take priority over --include-dir\n\+ patterns. GLOB should be quoted to prevent shell globbing. This\n\+ option may be repeated.\n\ --include-from=FILE\n\ Read the globs from FILE and search only files and directories\n\- whose name matches one or more globs. A glob can use **, *, ?, and\n\+ whose name matches one or more globs. A glob may use **, *, ?, and\n\ [...] as wildcards and \\ to quote a wildcard or backslash\n\ character literally. When a glob contains a `/', full pathnames\n\ are matched. Otherwise basenames are matched. When a glob ends\n\@@ -14514,7 +14348,7 @@ void help(std::ostream& out) -e PATTERN, i.e. a PATTERN argument requires option -e. Press F1\n\ or CTRL-Z to view the help screen. Press F2 or CTRL-Y to invoke a\n\ command to view or edit the file shown at the top of the screen.\n\- The command can be specified with option --view and defaults to\n\+ This command can be specified with option --view and defaults to\n\ environment variable PAGER when defined, or VISUAL or EDITOR.\n\ Press TAB or SHIFT-TAB to navigate directories and to select a file\n\ to search. Press ENTER to select lines to output. Press ALT-l for\n\@@ -14647,10 +14481,11 @@ void help(std::ostream& out) additional values are output. See also options --format and -u.\n\ -Y, --empty\n\ Empty-matching patterns match all lines. Normally, empty matches\n\- are not output, unless a pattern begins with `^' or ends with `$'.\n\- With this option, empty-matching patterns, such as x? and x*, match\n\- all lines, not only lines with an `x' (enabled by default in grep\n\- compatibility mode).\n\+ are not output, unless a pattern starts with `^' or ends with `$'.\n\+ With this option, empty-matching patterns, such as x*y?, x*y*z*,\n\+ and x*(y?|z) match all lines, not only the expected pattern.\n\+ This option is enabled by default in grep compatibility mode;\n\+ specify option --no-empty to prevent matching all lines.\n\ -y, --any-line, --passthru\n\ Any line is output (passthru). Non-matching lines are output as\n\ context with a `-' separator. See also options -A, -B and -C.\n\@@ -14683,8 +14518,8 @@ void help(std::ostream& out) with option --zmax=NUM, searches the contents of compressed files\n\ and archives stored within archives up to NUM levels. When -g, -O,\n\ -M, or -t is specified, searches archives for files that match the\n\- specified globs, file extensions, file signature magic bytes, or\n\- file types, respectively; a side-effect of these options is that\n\+ specified globs, filename extensions, file signature magic bytes,\n\+ or file types, respectively; a side-effect of these options is that\n\ the compressed files and archives searched are only those with\n\ filename extensions that match known compression and archive types.\n" #ifndef HAVE_LIBZ@@ -14698,19 +14533,19 @@ void help(std::ostream& out) #endif #ifdef HAVE_LIBBZ2 ",\n\- bzip2 (requires suffix .bz, .bz2, .bzip2, .tbz, .tbz2, .tb2, .tz2)"+ bzip2 (.bz, .bz2, .bzip2, .tbz, .tbz2, .tb2, .tz2)" #endif #ifdef HAVE_LIBLZMA ",\n\- lzma and xz (requires suffix .lzma, .tlz, .xz, .txz)"+ xz (.xz, .txz) and lzma (requires suffix .lzma, .tlz)" #endif-#ifdef HAVE_LIBLZ4+#ifdef HAVE_LIBZSTD ",\n\- lz4 (requires suffix .lz4)"+ zstd (.zst, .zstd, .tzst)" #endif-#ifdef HAVE_LIBZSTD+#ifdef HAVE_LIBLZ4 ",\n\- zstd (requires suffix .zst, .zstd, .tzst)"+ lz4 (requires suffix .lz4)" #endif #ifdef HAVE_LIBBROTLI ",\n\@@ -15022,6 +14857,7 @@ Gitignore-style globbing is performed by all glob-related options: -g (--glob),\ [^abc-e] one character not a,b,c,d,e,/\n\ [!abc-e] one character not a,b,c,d,e,/\n\ / when used at the start of a glob, matches working directory\n\+ when used at the end of a glob, matches directories only\n\ **/ zero or more directories\n\ /** when at the end of a glob, matches everything after the /\n\ \\? a ? or any other character specified after the backslash\n\@@ -15030,13 +14866,14 @@ A glob pattern starting with a ^ or a ! inverts the matching. Instead of\n\ matching a filename or directory name, the directory or file is ignored and\n\ excluded from the search.\n\ \n\-When a glob pattern contains a /, the full pathname is matched. Otherwise, the\n\-basename of a file or directory is matched in recursive searches.\n\+When a glob pattern contains a /, the full pathname is matched from the working\n\+directory down. Otherwise, the basename of a file or directory is matched in\n\+recursive searches.\n\ \n\-When a glob pattern begins with a /, files and directories are matched at the\n\-working directory, not recursively.\n\+When a glob pattern starts with ./ or just a single /, the glob is matched at\n\+the working directory.\n\ \n\-When a glob pattern ends with a /, only directories are matched, not files.\n\+When a glob pattern ends with a /, directories are matched.\n\ \n\ Option -g (--glob) performs glob matching with the specified glob pattern or\n\ a set of comma-separated globs. When a glob is preceded with a ^ or !, the\n\@@ -15113,7 +14950,7 @@ IMPORTANT:\n\ \n\ Fuzzy search anchors each pattern match at the beginning character of the\n\ specified regex pattern. This significantly improves search performance and\n\-prevents overmatching. For example, the regex pattern `[1-9]buzz' begins with\n\+prevents overmatching. For example, the regex pattern `[1-9]buzz' starts with\n\ a nonzero digit at which all matches are anchored. Replace the beginning of a\n\ regex pattern with a wildcard to match any character, for example `.buzz', or\n\ make the first characters of the regex pattern optional, for example\n\@@ -15194,9 +15031,7 @@ void is_directory(const char *pathname) void cannot_decompress(const char *pathname, const char *message) { if (!flag_no_messages)- { fprintf(Static::errout, "%sugrep: cannot decompress %s%s%s: %s\n", color_off, color_fn, pathname, color_off, message != NULL ? message : "");- } } #endif
@@ -38,7 +38,7 @@ #define UGREP_HPP // DO NOT ALTER THIS LINE: updated by makemake.sh and we need it physically here for MSVC++ build from source-#define UGREP_VERSION "7.3.0"+#define UGREP_VERSION "7.4.0" // disable mmap because mmap is almost always slower than the file reading speed improvements since 3.0.0 #define WITH_NO_MMAP@@ -49,8 +49,8 @@ // use a lock-free job queue, which is appears to be SLOWER than a standard simple lock-based queue for each worker // #define WITH_LOCK_FREE_JOB_QUEUE -// drain stdin until eof to prevent broken pipe signal-// #define WITH_STDIN_DRAIN+// drain stdin until eof to keep reading from stdin such as a pipe (and prevent broken pipe signal)+#define WITH_STDIN_DRAIN // quick warn about unreadable file/dir arguments before searching by checking stat S_IRUSR // #define WITH_WARN_UNREADABLE_FILE_ARG@@ -67,7 +67,7 @@ // #define WITH_XDG_CONFIG_HOME // check if we are compiling for a windows OS, but not Cygwin or MinGW-#if (defined(__WIN32__) || defined(_WIN32) || defined(WIN32) || defined(__BORLANDC__)) && !defined(__CYGWIN__) && !defined(__MINGW32__) && !defined(__MINGW64__)+#if (defined(__WIN32__) || defined(_WIN32) || defined(WIN32) || defined(_WIN64) || defined(__BORLANDC__)) && !defined(__CYGWIN__) && !defined(__MINGW32__) && !defined(__MINGW64__) # define OS_WIN #endif
@@ -41,6 +41,7 @@ #include <cstdlib> #include <cstdint> #include <cstring>+#include <cctype> #include <ctime> #include <climits> #include <exception>@@ -252,6 +253,11 @@ class zstreambuf : public std::streambuf { static const uint32_t ZIP_EMPTY_MAGIC = 0x06054b50; // zip empty archive header magic static const uint32_t ZIP_DESCRIPTOR_MAGIC = 0x08074b50; // zip descriptor magic + static const uint32_t ZSTD_HEADER_MAGIC = 0xfd2fb528; // zstd header magic++ static const uint32_t XZ_HEADER_MAGIC_1 = 0x587a37fd; // 4 bytes of xz header magic+ static const uint32_t XZ_HEADER_MAGIC_2 = 0x005a; // next 2 bytes of xz header magic+ // read zip local file header if we are at a header, read the header, file name, and extra field bool header() {@@ -954,7 +960,7 @@ class zstreambuf : public std::streambuf { zlen_ -= zcur_; memmove(zbuf_, zbuf_ + zcur_, zlen_); zcur_ = 0;- size_t ret = fread(zbuf_ + zlen_, 1, ZIPBLOCK - zlen_, file_);+ size_t ret = file_read(file_, zbuf_ + zlen_, ZIPBLOCK - zlen_); zlen_ += ret; if (zlib_strm_ != NULL) {@@ -999,7 +1005,7 @@ class zstreambuf : public std::streambuf { zlen_ -= zcur_; memmove(zbuf_, zbuf_ + zcur_, zlen_); zcur_ = 0;- size_t ret = fread(zbuf_ + zlen_, 1, ZIPBLOCK - zlen_, file_);+ size_t ret = file_read(file_, zbuf_ + zlen_, ZIPBLOCK - zlen_); zlen_ += ret; if (zlen_ >= num) {@@ -1018,7 +1024,7 @@ class zstreambuf : public std::streambuf { bool read() { zcur_ = 0;- zlen_ = fread(zbuf_, 1, ZIPBLOCK, file_);+ zlen_ = file_read(file_, zbuf_, ZIPBLOCK); return zlen_ > 0; } @@ -1328,8 +1334,8 @@ class zstreambuf : public std::streambuf { else if (is_bz3(pathname)) { #ifdef HAVE_LIBBZIP3- // try to read the compression format magic bytes- if (fread(buf_, 1, 9, file) < 9 || strncmp(reinterpret_cast<char*>(buf_), "BZ3v1", 5) != 0)+ // try to read the compression format magic bytes and block size+ if (file_read(file_, buf_, 9) < 9 || strncmp(reinterpret_cast<char*>(buf_), "BZ3v1", 5) != 0) { cannot_decompress(pathname_, "an error was detected in the bzip3 compressed data"); delete bz3file_;@@ -1413,7 +1419,7 @@ class zstreambuf : public std::streambuf { else { // try to read two compression format magic bytes- size_t num = fread(buf_, 1, 2, file);+ size_t num = file_read(file_, buf_, 2); if (num == 2 && u16(buf_) == ZipInfo::DEFLATE_HEADER_MAGIC) {@@ -1459,7 +1465,7 @@ class zstreambuf : public std::streambuf { else { // read up to four bytes of the compression format's magic bytes to check for zip- num += fread(buf_ + num, 1, 4 - num, file);+ num += file_read(file_, buf_ + num, 4 - num); if (num == 4 && u32(buf_) == ZipInfo::ZIP_HEADER_MAGIC) {@@ -1497,9 +1503,109 @@ class zstreambuf : public std::streambuf { } else {- // no compression: pass through- num += fread(buf_ + num, 1, Z_BUF_LEN - num, file);- len_ = static_cast<std::streamsize>(num);+#ifdef HAVE_LIBZSTD+ // compare 4 bytes to check for zstd+ if (num == 4 && u32(buf_) == ZipInfo::ZSTD_HEADER_MAGIC)+ {+ // open zstd compressed file and pass the first 4 bytes to its input buffer+ try+ {+ zstdfile_ = new ZSTD();+ if (zstdfile_->strm == NULL || zstdfile_->zbuf == NULL)+ {+ warning("ZSTD_createDStream failed", pathname);++ delete zstdfile_;+ zstdfile_ = NULL;+ file_ = NULL;+ }+ else+ {+ memcpy(zstdfile_->zbuf, buf_, 4);+ zstdfile_->zlen = 4;+ }+ }++ catch (const std::bad_alloc&)+ {+ cannot_decompress(pathname_, "out of memory");+ file_ = NULL;+ }+ }+ else+#endif+ {+#ifdef HAVE_LIBLZMA+ // read up to 6 bytes and compare to check for xz+ num += file_read(file_, buf_ + num, 6 - num);++ if (num == 6 && u32(buf_) == ZipInfo::XZ_HEADER_MAGIC_1 && u16(buf_ + 4) == ZipInfo::XZ_HEADER_MAGIC_2)+ {+ // open xz compressed file and pass the first 6 bytes to its input buffer+ try+ {+ xzfile_ = new XZ();+ xzfile_->strm.next_in = buf_;+ xzfile_->strm.avail_in = 6;+ lzma_ret ret = lzma_auto_decoder(&xzfile_->strm, UINT64_MAX, LZMA_TELL_UNSUPPORTED_CHECK | LZMA_CONCATENATED);+ if (ret != LZMA_OK)+ {+ warning("lzma_stream_decoder failed", pathname);++ delete xzfile_;+ xzfile_ = NULL;+ file_ = NULL;+ }+ }++ catch (const std::bad_alloc&)+ {+ cannot_decompress(pathname_, "out of memory");+ file_ = NULL;+ }+ }+ else+#endif+ {+#ifdef HAVE_LIBBZ2+ // read up to 10 bytes and compare to check for bzip2+ num += file_read(file_, buf_ + num, 10 - num);++ if (num == 10 && memcmp(buf_, "BZh", 3) == 0 && isdigit(buf_[3]) && memcmp(buf_ + 4, "\x31\x41\x59\x26\x53\x59", 6) == 0)+ {+ // open bzip2 compressed file+ try+ {+ bzfile_ = new BZ();+ bzfile_->strm.next_in = reinterpret_cast<char*>(buf_);+ bzfile_->strm.avail_in = 10;+ int ret = BZ2_bzDecompressInit(&bzfile_->strm, 0, 0);+ if (ret != BZ_OK)+ {+ warning("BZ2_bzDecompressInit failed", pathname);++ delete bzfile_;+ bzfile_ = NULL;+ file_ = NULL;+ }+ }++ catch (const std::bad_alloc&)+ {+ cannot_decompress(pathname_, "out of memory");+ file_ = NULL;+ }+ }+ else+#endif++ {+ // no compression: pass through+ num += file_read(file_, buf_ + num, Z_BUF_LEN - num);+ len_ = static_cast<std::streamsize>(num);+ }+ }+ } } } }@@ -1941,6 +2047,12 @@ class zstreambuf : public std::streambuf { #endif + // read from file into buffer, return len when successful or less when EOF reached or error+ static size_t file_read(FILE *file, unsigned char *buf, size_t len)+ {+ return fread(buf, 1, len, file);+ }+ // fetch and decompress the next block of data into buf[0..len-1], return number of bytes decompressed, zero on EOF or negative on error std::streamsize next(unsigned char *buf, size_t len) {@@ -1974,7 +2086,7 @@ class zstreambuf : public std::streambuf { // read compressed data into zfile_->zbuf[] and decompress zfile_->zbuf[] into the given buf[] if (ret == Z_OK && zfile_->strm.avail_in == 0 && num < static_cast<std::streamsize>(len) && !zfile_->zend) {- zfile_->zlen = fread(zfile_->zbuf, 1, Z_BUF_LEN, file_);+ zfile_->zlen = file_read(file_, zfile_->zbuf, Z_BUF_LEN); if (ferror(file_)) {@@ -2085,7 +2197,7 @@ class zstreambuf : public std::streambuf { // read compressed data into bzfile_->zbuf[] and decompress bzfile_->zbuf[] into the given buf[] if (ret == BZ_OK && bzfile_->strm.avail_in == 0 && num < static_cast<std::streamsize>(len) && !bzfile_->zend) {- bzfile_->zlen = fread(bzfile_->zbuf, 1, Z_BUF_LEN, file_);+ bzfile_->zlen = file_read(file_, bzfile_->zbuf, Z_BUF_LEN); if (ferror(file_)) {@@ -2166,7 +2278,7 @@ class zstreambuf : public std::streambuf { if (ret != LZMA_OK && ret != LZMA_STREAM_END) {- cannot_decompress(pathname_, "an error was detected in the lzma compressed data");+ cannot_decompress(pathname_, "an error was detected in the xz/lzma compressed data"); num = -1; } else@@ -2178,7 +2290,7 @@ class zstreambuf : public std::streambuf { // read compressed data into xzfile_->zbuf[] and decompress xzfile_->zbuf[] into the given buf[] if (ret == LZMA_OK && xzfile_->strm.avail_in == 0 && num < static_cast<std::streamsize>(len) && !xzfile_->zend) {- xzfile_->zlen = fread(xzfile_->zbuf, 1, Z_BUF_LEN, file_);+ xzfile_->zlen = file_read(file_, xzfile_->zbuf, Z_BUF_LEN); if (ferror(file_)) {@@ -2204,7 +2316,7 @@ class zstreambuf : public std::streambuf { if (ret != LZMA_OK && ret != LZMA_STREAM_END) {- cannot_decompress(pathname_, "an error was detected in the lzma compressed data");+ cannot_decompress(pathname_, "an error was detected in the xz/lzma compressed data"); num = -1; } else@@ -2260,7 +2372,7 @@ class zstreambuf : public std::streambuf { // frame header is 7 to 19 bytes long if (lz4file_->zlen < 19)- lz4file_->zlen += fread(lz4file_->zbuf + lz4file_->zlen, 1, Z_BUF_LEN, file_);+ lz4file_->zlen += file_read(file_, lz4file_->zbuf + lz4file_->zlen, Z_BUF_LEN); // if not enough frame data, error if not at EOF, otherwise EOF is OK if (lz4file_->zlen < 7)@@ -2295,7 +2407,7 @@ class zstreambuf : public std::streambuf { while (size > 0) {- size_t ret = fread(lz4file_->zbuf, 1, size < Z_BUF_LEN ? size : Z_BUF_LEN, file_);+ size_t ret = file_read(file_, lz4file_->zbuf, size < Z_BUF_LEN ? size : Z_BUF_LEN); if (ret == 0) break; size -= ret;@@ -2355,7 +2467,7 @@ class zstreambuf : public std::streambuf { // need 4 bytes with block size if (lz4file_->zlen < 4)- lz4file_->zlen += fread(lz4file_->zbuf + lz4file_->zlen, 1, Z_BUF_LEN, file_);+ lz4file_->zlen += file_read(file_, lz4file_->zbuf + lz4file_->zlen, Z_BUF_LEN); if (lz4file_->zlen < 4) { warning("cannot read", pathname_);@@ -2397,7 +2509,7 @@ class zstreambuf : public std::streambuf { // read the rest of the block, may overshoot by up to Z_BUF_LEN bytes while (lz4file_->zloc + size > lz4file_->zlen) {- size_t ret = fread(lz4file_->zbuf + lz4file_->zlen, 1, Z_BUF_LEN, file_);+ size_t ret = file_read(file_, lz4file_->zbuf + lz4file_->zlen, Z_BUF_LEN); if (ret == 0) break; lz4file_->zlen += ret;@@ -2445,7 +2557,7 @@ class zstreambuf : public std::streambuf { { // need 4 bytes with block checksum if (lz4file_->zloc + 4 > lz4file_->zlen)- lz4file_->zlen += fread(lz4file_->zbuf + lz4file_->zlen, 1, Z_BUF_LEN, file_);+ lz4file_->zlen += file_read(file_, lz4file_->zbuf + lz4file_->zlen, Z_BUF_LEN); if (lz4file_->zloc + 4 > lz4file_->zlen) { warning("cannot read", pathname_);@@ -2502,7 +2614,7 @@ class zstreambuf : public std::streambuf { if (num >= 0 && num < static_cast<std::streamsize>(len) && zstdfile_->zloc >= zstdfile_->zlen && !zstdfile_->zend) { zstdfile_->zloc = 0;- zstdfile_->zlen = fread(zstdfile_->zbuf, 1, ZSTD_DStreamInSize(), file_);+ zstdfile_->zlen = file_read(file_, zstdfile_->zbuf, ZSTD_DStreamInSize()); if (ferror(file_)) {@@ -2570,7 +2682,7 @@ class zstreambuf : public std::streambuf { // read compressed data into brfile_->zbuf[] and decompress brfile_->zbuf[] into the given buf[] if (ret != BROTLI_DECODER_RESULT_ERROR && brfile_->avail_in == 0 && num < static_cast<std::streamsize>(len) && !brfile_->zend) {- brfile_->zlen = fread(brfile_->zbuf, 1, Z_BUF_LEN, file_);+ brfile_->zlen = file_read(file_, brfile_->zbuf, Z_BUF_LEN); if (ferror(file_)) {@@ -2627,7 +2739,7 @@ class zstreambuf : public std::streambuf { while (bz3file_->loc >= bz3file_->len) { // read decompressed and compressed size- if (fread(bz3file_->buf, 1, 8, file_) < 8)+ if (file_read(file_, bz3file_->buf, 8) < 8) { if (ferror(file_)) {@@ -2647,7 +2759,7 @@ class zstreambuf : public std::streambuf { if (block_size > bz3file_->max || bz3file_->len > bz3file_->max ||- fread(bz3file_->buf, 1, block_size, file_) < block_size ||+ file_read(file_, bz3file_->buf, block_size) < block_size || bz3_decode_block(bz3file_->strm, bz3file_->buf, bz3file_->max, block_size, bz3file_->len) < 0) { if (ferror(file_))@@ -2694,10 +2806,10 @@ class zstreambuf : public std::streambuf { else if (file_ != NULL) { // pass through, without decompression- num = fread(buf, 1, len, file_);+ num = file_read(file_, buf, len); // end of file or error?- if (num < static_cast<std::streamsize>(len))+ if (num == 0 || (num < static_cast<std::streamsize>(len) && feof(file_))) file_ = NULL; }
tests/out/Hello_wnhS--replace.out
@@ -1,6 +1,5 @@ [1;35mHello.bat[m[1;36m:[m#!/bin/[m[1;4;32m(1:sh-echo)[m "Hello World!"[m-[1;35mHello.bat[m[1;36m:[mecho "[m[1;4;32m(2:Hello World!")[m+echo)[m "[m[1;4;32m(2:Hello World!")[m [mBinary file [1;35mHello.class[m matches [1;35mHello.java[m[1;36m:[m[1;4;32m(1:import java.lang.*;)[m [1;35mHello.java[m[1;36m:[m[1;4;32m(2:public class)[m [m[1;4;32m(3:Hello //)[m [m[1;4;32m(4:prints a)[m [m[1;4;32m(5:Hello World!)[m [m[1;4;32m(6:greeting@@ -8,5 +7,4 @@ echo)[m "Hello World!"[m [1;35mHello.java[m[1;36m:[m { System.out.println("[m[1;4;32m(9:Hello World!");)[m [mBinary file [1;35mHello.pdf[m matches [1;35mHello.sh[m[1;36m:[m#!/bin/[m[1;4;32m(1:sh-echo)[m "Hello World!"[m-[1;35mHello.sh[m[1;36m:[mecho "[m[1;4;32m(2:Hello World!")[m+echo)[m "[m[1;4;32m(2:Hello World!")[m