200120.1 · 8acadb0c492cf207ac06b07e92b1c008b289c6de
released 1.7.3 - auto-detecting compressed files with -z; new "negate" operators ! and ^ for file types, extensions, and signatures
reading declarations
@@ -7,27 +7,26 @@ Universal grep pro ("uber grep") <img src="https://www.genivia.com/images/function_defs.png" width="45%" height="45%" alt="ugrep C++ function search results"> <img src="https://www.genivia.com/images/hexdump.png" width="45%" height="45%" alt="ugrep hexdump results"> <br>-Grep super fast through source code, Unicode text, binary files, cpio/tar/pax/zip archives, and compressed files.+Grep super fast through source code, Unicode text, binary files, cpio/jar/pax/tar/zip archives, and compressed files. <br> <br> </div> - Unlimited pro version-- Written in clean and efficient C++11 code-- Built for extreme speed+- Fully compatible with the standard GNU and BSD grep command-line options+- Written in clean and efficient C++11 code, built for extreme speed - Faster than GNU/BSD grep, mostly beating ripgrep, silver searcher, etc. - Multi-threaded search using high-performance lock-free job queue stealing-- Task-parallel decompression and search of decompressed streams-- Compatible with the standard GNU and BSD grep command-line options-- Selects files to search by file types-- Selects files to search by filename extensions and "magic bytes"-- Searches archives (cpio, tar, pax, zip)+- Multi-threaded decompression and search of decompressed streams+- Optimized non-blocking asynchronous IO+- Selects files to search by file types, filename suffix, and/or "magic bytes"+- Searches archives (cpio, jar, tar, pax, zip) - Searches compressed files (.gz, .Z, .zip, .bz, .bz2, .lzma, .xz)-- Searches documents such as pdf, odt, doc, and docx using third-party filter utilities+- Searches pdf, doc, docx, xls, xlxs, and more using filter utilities+- Searches binary files and displays hexdumps with binary pattern matches - Searches UTF-encoded files by supporting Unicode pattern matches - Searches files encoded in ISO-8859-1 thru 16, CP 437, CP 850, MAC, KOI8, etc. - Searches files excluding files specified by .gitignore etc.-- Searches binary patterns and displays hexdumps with matches - Searches patterns across newlines - Searches patterns excluding negative patterns ("match this but not that") - Includes predefined regex patterns to search source code, XML, JSON, HTML, etc.@@ -111,13 +110,13 @@ Introduction: why use ugrep? options `-O` and `-M` may be used to select files by extension and by file signature "magic bytes", respectively. -- **ugrep searches compressed files and archives (cpio, tar, pax, and zip)**- with option `-z`. The matching file names in tar files are output in braces.+- **ugrep searches compressed files and archives (cpio, jar, tar, pax, zip)**+ with option `-z`. The matching file names in archives are output in braces. For example `myprojects.tgz{main.cpp}` indicates that file `main.cpp` in compressed tar file `myprojects.tgz` has a match. Filename glob matching, file types, filename extensions, and file signature "magic bytes" can be- selected to filter files in tar files with options `-g`, `-t`, `-O`, and- `-M`, respectively. For example:+ selected to filter files in archives with options `-g`, `-t`, `-O`, and `-M`,+ respectively. For example: ugrep -z -tc++ -w 'main' myprojects.tgz @@ -330,7 +329,7 @@ This builds `ugrep` in the `ugrep/src` directory and copy it to `ugrep/bin`. You can tell which version it is with: $ bin/ugrep -V- ugrep 1.7.2 x86_64-apple-darwin16.7.0+ ugrep 1.7.3 x86_64-apple-darwin16.7.0 If you prefer to enable colorized output by default without requiring option `--color` then execute:@@ -531,7 +530,8 @@ not support Unicode!): Option `-J1` specifies one thread of execution to produce the exact same output ordering as GNU/BSD grep at the cost of lower performance and `-Y` enables-empty matches for GNU/BSD compatibility, see [further below](#improvements).+empty matches for GNU/BSD compatibility (`-Y` is not strictly necessary, see+[further below](#improvements).) <a name="aliases"/> @@ -573,6 +573,8 @@ empty matches for GNU/BSD compatibility, see [further below](#improvements). GNU/BSD grep, pattern `a*` matches every line in the input, and actually matches `xyz` three times (the empty transitions before and between the `x`, `y`, and `z`). Allowing empty matches requires **ugrep** option `-Y`.+ Patterns that start with `^` and end with `$` are permitted to match empty,+ e.g. `^\h*$`, by implicitly enabling `-Y`. - **ugrep** offers *negative patterns* `-N PATTERN`, which are patterns of the form `(?^X)` to skip input that matches `X`. Negative patterns can be used to skip strings and comments when searching for identifiers in source code@@ -582,14 +584,14 @@ empty matches for GNU/BSD compatibility, see [further below](#improvements). - **ugrep** produces hexdumps with `-W` (output binary matches in hex with text matches output as usual) and `-X` (output all matches in hex). - **ugrep** searches compressed files with option `-z`.-- **ugrep** searches cpio, tar, pax, and zip archives with option `-z`.-- **ugrep** searches pdf, odt, doc, docx, epub, and so on with `--filter` using- third-party format conversion utilities as plugins.+- **ugrep** searches cpio, jar, pax, tar, and zip archives with option `-z`.+- **ugrep** searches pdf, doc, docx, xls, xlsx, epub, and more with `--filter`+ using third-party format conversion utilities as plugins. - **ugrep** options `-O`, `-M`, and `-t` specify file extensions, file signature magic byte patterns, and predefined file types, respectively. This allows searching for certain types of files in directory trees, for example with recursive search options `-R` and `-r`. Options `-O`, `-M`, and `-t`- also applies to archived files in cpio, tar, pax, and zip files.+ also applies to archived files in cpio, jar, pax, tar, and zip files. - **ugrep** option `-k`, `--column-number` to display the column number, taking tab spacing into account by expanding tabs, as specified by option `--tabs`. - **ugrep** option `-f` uses `GREP_PATH` environment variable or the predefined@@ -796,11 +798,11 @@ To show a list of `-t TYPES` option values: searched, the string ``(standard input)'' is written. -R, --dereference-recursive Recursively read all files under each directory. Follow all- symbolic links, unlike -r. If -J1 is specified, files are searched- in the same order as specified.+ symbolic links, unlike -r. When -J1 is specified, files are+ searched in the same order as specified. -r, --recursive Recursively read all files under each directory, following symbolic- links only if they are on the command line. If -J1 is specified,+ links only if they are on the command line. When -J1 is specified, files are searched in the same order as specified. To recursively list all non-empty files in the working directory, following@@ -857,7 +859,7 @@ To recursively list all shell scripts based on extensions only with `-tshell`: -N PATTERN, --not-regexp=PATTERN Specify a negative PATTERN used during the search of the input: an input line is selected only if it matches any of the specified- patterns when PATTERN does not match. Same as -e (?^PATTERN)+ patterns when PATTERN does not match. Same as -e (?^PATTERN). Negative PATTERN matches are removed before any other specified patterns are matched. Note that longer patterns take precedence over shorter patterns. This option may be repeated.@@ -865,18 +867,14 @@ To recursively list all shell scripts based on extensions only with `-tshell`: Selected lines are those not matching any of the specified patterns. -w, --word-regexp- The PATTERN or -e PATTERN is searched for as a word (as if- surrounded by \< and \>). If PATTERN or -e PATTERN is also- specified, then this option does not apply to -f FILE patterns.+ The PATTERN is searched for as a word (as if surrounded by \< and+ \>). If a PATTERN is specified (or -e PATTERN or -N PATTERN), then+ this option does not apply to -f FILE patterns. -x, --line-regexp- Only input lines selected against the entire PATTERN or -e PATTERN- is considered to be matching lines (as if surrounded by ^ and $).- If PATTERN or -e PATTERN is also specified, then this option does+ Only input lines selected against the entire PATTERN is considered+ to be matching lines (as if surrounded by ^ and $). If a PATTERN+ is specified (or -e PATTERN or -N PATTERN), then this option does not apply to -f FILE patterns.- -Y, --empty- Permits empty matches, such as `^\h*$' to match blank lines. Empty- matches are disabled by default. Note that empty-matching patterns,- such as x? and x*, match all input with -Y, not only lines with `x'. To display lines in file `myfile.sh` but not lines matching `^[ \t]*#`: @@ -902,9 +900,9 @@ To display lines that are not blank lines: ugrep -x '.*' -N '\h*' myfile.py -Same, but using `-v` and `-x`, and by using `-Y` to match empty patterns:+Same, but using `-v` and `-x` with `\h*`, i.e. pattern `^\h*$`: - ugrep -v -x -Y '\h*' myfile.py+ ugrep -v -x '\h*' myfile.py To recursively list all Python files that do not contain the word `display`, allowing the word to occur in strings and comments:@@ -1033,7 +1031,7 @@ To search file `spanish-iso.txt` encoded in ISO-8859-1: ### Matching multiple lines of text -o, --only-matching- Prints only the matching part of lines. When multiple lines match,+ Print only the matching part of lines. When multiple lines match, the line numbers with option -n are displayed using `|' as the field separator for each additional line matched by the pattern. This option cannot be combined with options -A, -B, -C, -v, and -y.@@ -1131,19 +1129,32 @@ characters of context instead of a single word: If that fails, looks for FILE in /usr/local/share/ugrep/pattern. When FILE is a `-', standard input is read. This option may be repeated.+ --ignore-files[=FILE]+ Ignore files and directories matching the globs in each FILE when+ encountered in recursive searches. The default FILE is+ `.gitignore'. Matching files and directories located in the+ directory tree rooted at a FILE's location are ignored by+ temporarily overriding the --exclude and --exclude-dir globs.+ Files and directories specified as FILE arguments are not ignored.+ This option may be repeated. -O EXTENSIONS, --file-extensions=EXTENSIONS- Search only files whose file name extensions match the specified- comma-separated list of file name EXTENSIONS. This option is the- same as specifying --include='*.ext' for each extension name `ext'- in the EXTENSIONS list. This option may be repeated and may be- combined with options -M and -t to expand the search.+ Search only files whose filename extensions match the specified+ comma-separated list of EXTENSIONS, same as --include='*.ext' for+ each `ext' in EXTENSIONS. When `ext' is preceded by a `!' or a+ `^', skip files whose filename extensions matches `ext', same as+ --exclude='*.ext'. This option may be repeated and may be combined+ with options -M and -t to expand the recursive search. -t TYPES, --file-type=TYPES Search only files associated with TYPES, a comma-separated list of- file types. Each file type corresponds to a set of file name+ file types. Each file type corresponds to a set of filename extensions passed to option -O. For capitalized file types, the- search is expanded to include files found on the search path with- matching file signature magic bytes passed to option -M. This+ search is expanded to include files with matching file signature+ magic bytes, as if passed to option -M. When a type is preceeded+ by a `!' or a `^', excludes files of the specified type. This option may be repeated.+ --stats+ Display statistics on the number of files and directories searched.+ Display the inclusion and exclusion constraints applied. The file types are listed with `ugrep -tlist`. The list is based on established filename extensions and "magic bytes". If you have a file type@@ -1161,6 +1172,20 @@ numbers with `-Oc,cpp`, `-o`, `-n`, and `-f c++/functions`: ugrep -R -on -Oc,cpp -f c++/functions +To recursively list all shell files with `-tShell` to match filename extensions+and files with shell shebangs, except files with suffix `.sh`:++ ugrep -Rl -tShell -O^sh ''++To recursively list all non-shell files with `-t^Shell`:++ ugrep -Rl -t^Shell ''++To recursively list all shell files with shell shebangs that have no shell+filename extensions:++ ugrep -Rl -tShell -t^shell ''+ To search for lines with `FIXME` in C/C++ comments, excluding `FIXME` in multi-line strings: @@ -1186,24 +1211,22 @@ comments: -z, --decompress Decompress files to search, when compressed. Archives (.cpio,- .pax, .tar, .zip) and compressed archives (e.g. .taz, .tgz, .tpz,- .tbz, .tbz2, .tb2, .tz2, .tlz, and .txz) are searched and matching- pathnames of files in archives are output in braces. If -g, -O,- -M, or -t is specified, searches files within archives whose name- matches globs, matches file name extensions, matches file- signature magic bytes, or matches file types, respectively.- Option --label=.ext associates suffix `ext' with standard input.- Supported compression formats: gzip (default, optional suffix .gz),- compress (requires suffix .Z), zip (requires suffix .zip or .ZIP),- lzma (requires suffix .lzma), xz (requires suffix .xz),- bzip2 (requires suffix .bz, .bz2, or .bzip2)+ .jar, .pax, .tar, .zip) and compressed archives (e.g. .taz, .tgz,+ .tpz, .tbz, .tbz2, .tb2, .tz2, .tlz, and .txz) are searched and+ matching pathnames of files in archives are output in braces. If+ -g, -O, -M, or -t is specified, searches files within archives+ whose name matches globs, matches file name extensions, matches+ file signature magic bytes, or matches file types, respectively.+ Supported compression formats: gzip (.gz), compress (.Z), zip,+ bzip2 (requires suffix .bz, .bz2, .bzip2, .tbz, .tbz2, .tb2, .tz2),+ lzma and xz (requires suffix .lzma, .tlz, .xz, .txz). Compressed files with gzip (`.gz`), compress (`.Z`), bzip2 (`.bz`, `.bz2`, `.bzip2`), lzma (`.lzma`), and xz (`.xz`) are searched with option `-z`. This option does not require files to be compressed. Uncompressed files are searched also. -Archives (cpio, tar, pax, and zip) are searched with option `-z`. Regular+Archives (cpio, jar, pax, tar, and zip) are searched with option `-z`. Regular files in an archive that match are output with the archive pathnames enclosed in `{` and `}` braces. Supported tar formats are v7, ustar, gnu, oldgnu, and pax. Supported cpio formats are odc, newc, and crc. Not supported is the@@ -1211,32 +1234,34 @@ obsolete non-portable old binary cpio format. Archive formats cpio, tar, and pax are automatically recognized with option `-z` based on their content, independent of their filename suffix. -The gzip/zlib format is automatically detected, which is useful when reading-gzip-compressed data from standard input. Other compression formats require a-filename suffix: `.Z` for compress, `.zip` or `.ZIP` for zip, `.bz`, `.bz2`, or-`.bzip2` for bzip2, `.lzma` for lzma, and `.xz` for xz. Also the compressed-tar archive shorthands `.taz`, `.tgz`, and `.tpz` for gzip, `.tbz`, `.tbz2`,-`.tb2`, and `.tz2` for bzip2, `.tlz` for lzma, and `.txz` for xz are+The gzip, compress, and zip formats are automatically detected, which is useful+when reading gzip-compressed data from standard input, e.g. input redirected+from a pipe. Other compression formats require a filename suffix: `.bz`,+`.bz2`, or `.bzip2` for bzip2, `.lzma` for lzma, and `.xz` for xz. Also the+compressed tar archive shorthands `.taz`, `.tgz`, and `.tpz` for gzip, `.tbz`,+`.tbz2`, `.tb2`, and `.tz2` for bzip2, `.tlz` for lzma, and `.txz` for xz are recognized. To decompress these formats from standard input, use option-`--label='stdin.Z'` for compress, `--label=stdin.zip` for zip, `--label='stdin.bz2'` for bzip2, `--label='stdin.lzma'` for lzma, and `--label='stdin.xz'` for xz. The name `stdin` is arbitrary and may be omitted: -format | filename suffix | tar/pax archive short suffix | suffix required? | ugrep stdin | lib required |+format | filename suffix | tar/pax archive short suffix | suffix required? | ugrep from stdin | lib required | --------- | ----------------------- | ------------------------------- | ---------------- | ---------------- | ------------ |-gzip/zlib | `.gz` | `.taz`, `.tgz`, `.tpz` | no | automatic | libz |-compress | `.Z` | | yes | `--label=.Z` | *built-in* |-zip | `.zip`, `.ZIP` | | yes | `--label=.zip` | libz+libbz2 |+gzip | `.gz` | `.taz`, `.tgz`, `.tpz` | no | automatic | libz |+compress | `.Z` | | no | automatic | *built-in* |+zip | `.zip`, `.ZIP` | | no | automatic | libz | bzip2 | `.bz`, `.bz2`, `.bzip2` | `.tb2`, `.tbz`, `.tbz2`, `.tz2` | yes | `--label=.bz2` | libbz2 | lzma | `.lzma` | `.tlz` | yes | `--label=.lzma` | liblzma | xz | `.xz` | `.txz` | yes | `--label=.xz` | liblzma | +The gzip, bzip2, and xz formats support concatenated compressed files.+Concatenated compressed files are searched as one file.+ Supported zip compression methods are stored (0), deflate (8), bzip2 (12) if libbz2 is available, lzma (14) and xz (95) if liblzma is available. Archives-compressed within zip archives are searched: all cpio, tar, and pax files in+compressed within zip archives are searched: all cpio, pax, and tar files in zip archives are automatically recognized and searched. Compressed files in archives are not recognized and searched however, such as any compressed files-stored in cpio, tar, pax, and zip archives, which are considered binary files.+stored in cpio, pax, tar, and zip archives, which are considered binary files. Searching encrypted zip archives is not supported (perhaps in future releases, depending on requests). @@ -1244,18 +1269,17 @@ When option `-z` is used with options `-g`, `-O`, `-M`, or `-t`, archives and compressed and uncompressed files that match the filename selection criteria (glob, extension, magic bytes, or file type) are searched only. For example, `ugrep -r -z -tc++` searches C++ files such as `main.cpp`, and also-`main.cpp.gz` and `main.cpp.xz` when present. Also any cpio, tar, pax, and zip+`main.cpp.gz` and `main.cpp.xz` when present. Also any cpio, pax, tar, and zip archives when present are searched for C++ files such as `main.cpp`. Use option `--stats` to see a list of the glob patterns applied to filter file pathnames in the recursive search and when searching archive contents. When option `-z` is used with options `-g`, `-O`, `-M`, or `-t` to search cpio,-tar, pax, and zip archives, archived files that match the filename selection+jar, pax, tar, and zip archives, archived files that match the filename selection criteria are searched only. -Option `-z` uses task parallelism to speed up searching and may produce results-for uncompressed files even faster than without option `-z`, depending on the-OS and hardware.+Option `-z` uses thread task parallelism to speed up searching by running the+decompressor while searching the decompressed stream concurrently. To recursively search C++ files including compressed files for the word `my_function`, while skipping C and C++ comments:@@ -1302,30 +1326,51 @@ search the files: ### Find files by file signature and shebang "magic bytes" with -M and -t + --ignore-files[=FILE]+ Ignore files and directories matching the globs in each FILE when+ encountered in recursive searches. The default FILE is+ `.gitignore'. Matching files and directories located in the+ directory tree rooted at a FILE's location are ignored by+ temporarily overriding the --exclude and --exclude-dir globs.+ Files and directories specified as FILE arguments are not ignored.+ This option may be repeated. -M MAGIC, --file-magic=MAGIC Only files matching the signature pattern MAGIC are searched. The signature \"magic bytes\" at the start of a file are compared to- the `MAGIC' regex pattern. When matching, the file will be- searched. This option may be repeated and may be combined with- options -O and -t to expand the search. This option is relatively- slow as every file on the search path is read to compare MAGIC.+ the MAGIC regex pattern. When matching, the file will be searched.+ When MAGIC is preceded by a `!' or a `^', skip files with matching+ MAGIC signatures. This option may be repeated and may be combined+ with options -O and -t to expand the search. Every file on the+ search path is read, making searches potentially more expensive. -t TYPES, --file-type=TYPES Search only files associated with TYPES, a comma-separated list of- file types. Each file type corresponds to a set of file name+ file types. Each file type corresponds to a set of filename extensions passed to option -O. For capitalized file types, the- search is expanded to include files found on the search path with- matching file signature magic bytes passed to option -M. This+ search is expanded to include files with matching file signature+ magic bytes, as if passed to option -M. When a type is preceeded+ by a `!' or a `^', excludes files of the specified type. This option may be repeated.+ --stats+ Display statistics on the number of files and directories searched.+ Display the inclusion and exclusion constraints applied.++To recursively list all files that start with `#!` shebangs:++ ugrep -Rl -M'#!' '' -To recursively list all files that start with `#!` shebangs with `-M'#!.*'`:+To recursively list all files that start with `#` but not with `#!` shebangs: - ugrep -Rl -M'#!.*' ''+ ugrep -Rl -M'#' -M'^#!' '' To recursively list all Python files (extension `.py` or a shebang) with `-tPython`: ugrep -Rl -tPython '' +To recursively list all non-shell files with `-t^Shell`:++ ugrep -Rl -t^Shell ''+ To list Python files (extension `.py` or a shebang) that have import statements, excluding hidden files: @@ -1342,12 +1387,11 @@ statements, excluding hidden files: `command' is a filter utility. The filter utility should read from standard input and write to standard output. Files matching one of `exts` are filtered only. One or more `option' separated by- spacing may be specified, which are passed verbatim to the command- without expansion. A `%' as `option' expands into the pathname,- which is the conventional dash `-' when searching standard input.- Option --label=.ext may be used to force filtering standard input.- For example, --filter='pdf:pdftotext % -' searches PDF files.-+ spacing may be specified, which are passed verbatim to the command.+ A `%' as `option' expands into the pathname to search. For example,+ --filter='pdf:pdftotext % -' searches PDF files. The `%' expands+ into a `-' when searching standard input. Option --label=.ext may+ be used to specify extenion `ext' when searching standard input. Filter utilities may be associated with specific filename extensions. A filter utility is selected based on the filename extension and executed by@@ -1358,7 +1402,7 @@ expanded pathname. When a utility is not found an error message is displayed. When a utility fails to produce output, e.g. when the specified options for the utility are invalid, the search is silently skipped. -**Warning:** option `--format` should not be used with utilities that modify+**Warning:** option `--filter` should not be used with utilities that modify files. Otherwise searches may be unpredicatable. In the worst case files may be lost, for example when the specified utility replaces or deletes the file passed to the command with `--filter` option `%`.@@ -1420,8 +1464,16 @@ To search .docx, .xlsx, and .pptx files converted to XML for a match with ugrep -lr -Odocx,xlsx,pptx --filter='docx,xlsx,pptx:unzip -p %' '10/6' -**Important:** unzipping .docx, .xlxs, .pptx produces extensive XML output-containing meta information and binary data such as images.+**Important:** unzipping docx, xlxs, pptx files produces extensive XML output+containing meta information and binary data such as images. By contrast,+**ugrep** option `-z` with `-Oxml` selects the XML components only:++ ugrep -z -lr -Odocx,xlsx,pptx,xml '10/6'++**Note:** docx, xlsx, and pptx are zip files containing multiple components.+When selecting the XML components with option `-Oxml` in docx, xlsx, and pptx+documents, we should also specify `-Odocx,xlsx,pptx` to search these type of+files, otherwise these files will be ignored. <a name="binary"/> @@ -1463,7 +1515,7 @@ option does not permit matching patterns with newlines): ugrep --color -UX -y '\x00+' myfile -To match the binary pattern `A3hhhhA3hh` (hex) in a binary file without+To match the binary pattern `A3..A3.` (hex) in a binary file without Unicode pattern matching (which would otherwise match `\xaf` as a Unicode character U+00A3 with UTF-8 byte sequence C2 A3) and display the results in hex with `-X` with pager `less -R`:@@ -1512,19 +1564,23 @@ source): ### Ignoring .gitignore-specified files with --ignore-files --ignore-files[=FILE]- Ignore files and directories specified in a FILE when encountered- in recursive searches. The default is `.gitignore'. Files and- directories matching the globs in FILE are ignored in the directory- tree rooted at each FILE's location by temporarily overriding- --exclude and --exclude-dir globs. This option may be repeated.+ Ignore files and directories matching the globs in each FILE when+ encountered in recursive searches. The default FILE is+ `.gitignore'. Matching files and directories located in the+ directory tree rooted at a FILE's location are ignored by+ temporarily overriding the --exclude and --exclude-dir globs.+ Files and directories specified as FILE arguments are not ignored.+ This option may be repeated. -Option `--ignore-files` looks for `.gitignore`, or the specified FILE argument-of this option, in recursive searches. When found, the `.gitignore` file is-used to exclude the files and directories matching the globs in `.gitignore` in-the directory tree rooted at the `.gitignore` location by temporarily-overriding the `--exclude` and `--exclude-dir` globs, i.e. the `.gitignore`-exclusions are applied precisely. Use `--stats` to show the selection criteria-applied to the search results.+Option `--ignore-files` looks for `.gitignore`, or the specified `FILE`, in+recursive searches. When found, the `.gitignore` file is used to exclude the+files and directories matching the globs in `.gitignore` in the directory tree+rooted at the `.gitignore` location by temporarily overriding the `--exclude`+and `--exclude-dir` globs, i.e. the `.gitignore` exclusions are applied+precisely and exclusively. Use `--stats` to show the selection criteria+applied to the search results and the locations of each `FILE` found. To avoid+confusion, files and directories specified as command-line arguments to+**ugrep** are never ignored. See also [Using gitignore-style globs to select directories and files to search](#gitignore). @@ -1570,16 +1626,17 @@ implicit: --exclude=GLOB Skip files whose name matches GLOB using wildcard matching, same as -g !GLOB. GLOB can use **, *, ?, and [...] as wildcards, and \ to- quote a wildcard or backslash character literally. If GLOB- contains /, full pathnames are matched. Otherwise basenames are- matched. Note that --exclude patterns take priority over --include- patterns. This option may be repeated.+ quote a wildcard or backslash character literally. When GLOB+ contains a `/', full pathnames are matched. Otherwise basenames+ are matched. Note that --exclude patterns take priority over+ --include patterns. This option may be repeated. --exclude-dir=GLOB Exclude directories whose name matches GLOB from recursive- searches. If GLOB contains /, full pathnames are matched.- Otherwise basenames are matched. Note that --exclude-dir patterns- take priority over --include-dir patterns. This option may be- repeated.+ searches. GLOB can use **, *, ?, and [...] as wildcards, and \\ to+ quote a wildcard or backslash character literally. When GLOB+ contains a `/', full pathnames are matched. Otherwise basenames+ are matched. Note that --exclude-dir patterns take priority over+ --include-dir patterns. This option may be repeated. --exclude-from=FILE Read the globs from FILE and skip files and directories whose name matches one or more globs (as if specified by --exclude and@@ -1588,8 +1645,8 @@ implicit: option may be repeated. -g GLOB, --glob=GLOB Search only files whose name matches GLOB, same as --include=GLOB.- If GLOB is preceded by a `!', skip files whose name matches GLOB,- same as --exclude=GLOB.+ When GLOB is preceded by a `!' or a `^', skip files whose name+ matches GLOB, same as --exclude=GLOB. --ignore-files[=FILE] Ignore files and directories specified in a FILE when encountered in recursive searches. The default is `.gitignore'. Files and@@ -1600,15 +1657,16 @@ implicit: Search only files whose name matches GLOB using wildcard matching, same as -g GLOB. GLOB can use **, *, ?, and [...] as wildcards, and \ to quote a wildcard or backslash character literally. If- GLOB contains /, file pathnames are matched. Otherwise file+ GLOB contains a `/', full pathnames are matched. Otherwise basenames are matched. Note that --exclude patterns take priority over --include patterns. This option may be repeated. --include-dir=GLOB Only directories whose name matches GLOB are included in recursive- searches. If GLOB contains /, full pathnames are matched.- Otherwise basenames are matched. Note that --exclude-dir patterns- take priority over --include-dir patterns. This option may be- repeated.+ searches. GLOB can use **, *, ?, and [...] as wildcards, and \\ to+ quote a wildcard or backslash character literally. When GLOB+ contains a `/', full pathnames are matched. Otherwise basenames+ are matched. Note that --exclude-dir patterns take priority over+ --include-dir patterns. This option may be repeated. --include-from=FILE Read the globs from FILE and search only files and directories whose name matches one or more globs (as if specified by --include@@ -1616,11 +1674,15 @@ implicit: FILE are ignored. When FILE is a `-', standard input is read. This option may be repeated. -O EXTENSIONS, --file-extensions=EXTENSIONS- Search only files whose file name extensions match the specified- comma-separated list of file name EXTENSIONS. This option is the- same as specifying --include='*.ext' for each extension name `ext'- in the EXTENSIONS list. This option may be repeated and may be- combined with options -M and -t to expand the search.+ Search only files whose filename extensions match the specified+ comma-separated list of EXTENSIONS, same as --include='*.ext' for+ each `ext' in EXTENSIONS. When `ext' is preceded by a `!' or a+ `^', skip files whose filename extensions matches `ext', same as+ --exclude='*.ext'. This option may be repeated and may be combined+ with options -M and -t to expand the recursive search.+ --stats+ Display statistics on the number of files and directories searched.+ Display the inclusion and exclusion constraints applied. See also [Including or excluding mounted file systems from searches](#fs). @@ -1714,21 +1776,22 @@ ignored by one or more .gitignore files, when any are present: ### Including or excluding mounted file systems from searches --exclude-fs=MOUNTS- Exclude file systems with mounting points MOUNTS from recursive- searches, MOUNTS is a comma-separated list of pathnames. Note- that --exclude-fs mounts take priority over --include-fs mounts.- This option may be repeated.- --include-fs=MOUNTS- Only file systems with mounting points MOUNTS are included in- recursive searches. MOUNTS is a comma-separated list of pathnames.- --include-fs=. restricts recursive searches to the file system of- the working directory only. Note that --exclude-fs mounts take+ Exclude file systems specified by MOUNTS from recursive searches,+ MOUNTS is a comma-separated list of mount points or pathnames of+ directories on file systems. Note that --exclude-fs mounts take priority over --include-fs mounts. This option may be repeated.+ --include-fs=MOUNTS+ Only file systems specified by MOUNTS are included in recursive+ searches. MOUNTS is a comma-separated list of mount points or+ pathnames of directories on file systems. --include-fs=. restricts+ recursive searches to the file system of the working directory+ only. Note that --exclude-fs mounts take priority over+ --include-fs mounts. This option may be repeated. These options control recursive searches across file systems. Mounted devices-and symbolic links to files and directories located on mounted devices may be-included or excluded from recursive searches by specifying a mount point or a-directory pathname located in the file system to identify the file system.+and symbolic links to files and directories located on mounted file systems may+be included or excluded from recursive searches by specifying a mount point or+a pathname of any directory on the file system to specify the file system. A list of mounted file systems is typically stored in `/etc/mtab`. @@ -1971,16 +2034,16 @@ To search tarballs for matching names of PDF files (assuming bash is our shell): ### Output matches in JSON, XML, CSV, C++ --cpp- Output file matches in C++. See also option --format.+ Output file matches in C++. See also the --format and -u options. --csv Output file matches in CSV. When option -H, -n, -k, or -b is used- additional CSV values are output. See also option --format.+ additional values are output. See also the --format and -u options. --json Output file matches in JSON. When option -H, -n, -k, or -b is used- additional JSON properties are output. See also option --format.+ additional values are output. See also the --format and -u options. --xml Output file matches in XML. When option -H, -n, -k, or -b is used- additional XML attributes are output. See also option --format.+ additional values are output. See also the --format and -u options. To recursively search for lines with `TODO` and display C++ file matches in JSON with line number properties:@@ -2036,7 +2099,7 @@ field | output `%[SEP]$` | set field separator to `SEP` for the rest of the format fields `%f` | the file pathname `%h` | the quoted file pathname-`%z` | the pathname in a (compressed) tar file+`%z` | the pathname in a (compressed) archive `%n` | the line number of the match `%k` | the column number of the match `%b` | the byte offset of the match@@ -2226,21 +2289,25 @@ line of context: ### Matching empty patterns with -Y -Y, --empty- Permits empty matches, such as `^\h*$' to match blank lines. Empty- matches are disabled by default. Note that empty-matching patterns,- such as x? and x*, match all input with -Y, not only lines with `x'.+ Permits empty matches. By default, empty matches are disabled,+ unless a pattern starts with `^' and ends with `$'. Note that -Y+ when specified with an empty-matching pattern such as x? and x*,+ match all input, not only lines with a `x'.++Option `-Y` permits empty pattern matches. This option is introduced by+**ugrep** to prevent accidental matching with empty patterns: empty-matching+patterns such as `x?` and `x*` match all input, not only lines with `x`. By+default, without `-Y`, patterns match lines with at least one `x` as intended. -Option `-Y` permits empty pattern matches, which is useful with certain anchors-such as `^` and `$` (the option is automatically enabled for pattern `^$` that-matches empty lines). This option is introduced by **ugrep** to prevent-accidental matching with empty patterns: empty-matching patterns such as `x?`-and `x*` match all input, not only lines with `x`. By default, without `-Y`,-patterns match lines with at least one `x` as intended.+This option is automatically enabled when a pattern starts with `^` and ends+with `$` is specified. For example, `^\h*$` matches blank lines, including+empty lines. -To recursively list files in the working directory with empty and blank lines,-i.e. lines with white space only and empty lines, with `-Y`:+To recursively list files in the working directory with blank lines, i.e. lines+with white space only, including empty lines (note that option `-Y` is+implicitly enabled since the pattern starts with `^` and ends with `$`): - ugrep -RlY '^\h*$'+ ugrep -Rl '^\h*$' <a name="tips"/> @@ -2375,8 +2442,8 @@ in markdown: encoding. See also the -a, -I, -U, -W, and -X options. --break- Groups matches per file and adds a line break between results- from different files.+ Group matches per file. Adds a header and line break between+ results from different files. -C[NUM], --context[=NUM] Print NUM lines of leading and trailing context surrounding each@@ -2401,56 +2468,58 @@ in markdown: rated list of ANSI SGR parameters. COLORS selectively overrides the colors specified by the GREP_COLORS environment variable. - --cpp Output file matches in C++. See also option --format.+ --cpp Output file matches in C++. See also the --format and -u+ options. - --csv Output file matches in CSV. When option -H, -n, -k, or -b is- used additional CSV values are output. See also option --for-- mat.+ --csv Output file matches in CSV. When option -H, -n, -k, or -b is+ used additional values are output. See also the --format and -u+ options. -D ACTION, --devices=ACTION- If an input file is a device, FIFO or socket, use ACTION to- process it. By default, ACTION is `skip', which means that+ If an input file is a device, FIFO or socket, use ACTION to+ process it. By default, ACTION is `skip', which means that devices are silently skipped. If ACTION is `read', devices read just as if they were ordinary files. -d ACTION, --directories=ACTION- If an input file is a directory, use ACTION to process it. By- default, ACTION is `read', i.e., read directories just as if- they were ordinary files. If ACTION is `skip', silently skip- directories. If ACTION is `recurse', read all files under each- directory, recursively, following symbolic links only if they- are on the command line. This is equivalent to the -r option.- If ACTION is `dereference-recurse', read all files under each+ If an input file is a directory, use ACTION to process it. By+ default, ACTION is `read', i.e., read directories just as if+ they were ordinary files. If ACTION is `skip', silently skip+ directories. If ACTION is `recurse', read all files under each+ directory, recursively, following symbolic links only if they+ are on the command line. This is equivalent to the -r option.+ If ACTION is `dereference-recurse', read all files under each directory, recursively, following symbolic links. This is equivalent to the -R option. -E, --extended-regexp- Interpret patterns as extended regular expressions (EREs). This+ Interpret patterns as extended regular expressions (EREs). This is the default. -e PATTERN, --regexp=PATTERN- Specify a PATTERN used during the search of the input: an input- line is selected if it matches any of the specified patterns.+ Specify a PATTERN used during the search of the input: an input+ line is selected if it matches any of the specified patterns. Note that longer patterns take precedence over shorter patterns.- This option is most useful when multiple -e options are used to- specify multiple patterns, when a pattern begins with a dash- (`-'), to specify a pattern after option -f or after the FILE+ This option is most useful when multiple -e options are used to+ specify multiple patterns, when a pattern begins with a dash+ (`-'), to specify a pattern after option -f or after the FILE arguments. --exclude=GLOB Skip files whose name matches GLOB using wildcard matching, same as -g !GLOB. GLOB can use **, *, ?, and [...] as wildcards, and- \ to quote a wildcard or backslash character literally. If GLOB- contains /, full pathnames are matched. Otherwise basenames are- matched. Note that --exclude patterns take priority over- --include patterns. This option may be repeated.+ \ to quote a wildcard or backslash character literally. When+ GLOB contains a `/', full pathnames are matched. Otherwise+ basenames are matched. Note that --exclude patterns take prior-+ ity over --include patterns. This option may be repeated. --exclude-dir=GLOB- Exclude directories whose name matches GLOB from recursive- searches. If GLOB contains /, full pathnames are matched. Oth-- erwise basenames are matched. Note that --exclude-dir patterns- take priority over --include-dir patterns. This option may be- repeated.+ Exclude directories whose name matches GLOB from recursive+ searches. GLOB can use **, *, ?, and [...] as wildcards, and \+ to quote a wildcard or backslash character literally. When GLOB+ contains a `/', full pathnames are matched. Otherwise basenames+ are matched. Note that --exclude-dir patterns take priority+ over --include-dir patterns. This option may be repeated. --exclude-from=FILE Read the globs from FILE and skip files and directories whose@@ -2460,43 +2529,44 @@ in markdown: This option may be repeated. --exclude-fs=MOUNTS- Exclude file systems with mounting points MOUNTS from recursive- searches, MOUNTS is a comma-separated list of pathnames. Note- that --exclude-fs mounts take priority over --include-fs mounts.+ Exclude file systems specified by MOUNTS from recursive+ searches, MOUNTS is a comma-separated list of mount points or+ pathnames of directories on file systems. Note that+ --exclude-fs mounts take priority over --include-fs mounts. This option may be repeated. -F, --fixed-strings- Interpret pattern as a set of fixed strings, separated by new-- lines, any of which is to be matched. This makes ugrep behave+ Interpret pattern as a set of fixed strings, separated by new-+ lines, any of which is to be matched. This makes ugrep behave as fgrep. If PATTERN or -e PATTERN is also specified, then this option does not apply to -f FILE patterns. -f FILE, --file=FILE- Read one or more newline-separated patterns from FILE. Empty- pattern lines in FILE are not processed. If FILE does not+ Read one or more newline-separated patterns from FILE. Empty+ pattern lines in FILE are not processed. If FILE does not exist, the GREP_PATH environment variable is used as the path to- FILE. If that fails, looks for FILE in- /usr/local/share/ugrep/patterns. When FILE is a `-', standard+ FILE. If that fails, looks for FILE in+ /usr/local/share/ugrep/patterns. When FILE is a `-', standard input is read. Empty files contain no patterns, thus nothing is matched. This option may be repeated. --filter=COMMANDS Filter files through the specified COMMANDS first before search- ing. COMMANDS is a comma-separated list of `exts:command- [option ...]', where `exts' is a comma-separated list of file-- name extensions and `command' is a filter utility. The filter- utility should read from standard input and write to standard+ [option ...]', where `exts' is a comma-separated list of file-+ name extensions and `command' is a filter utility. The filter+ utility should read from standard input and write to standard output. Files matching one of `exts` are filtered only. One or- more `option' separated by spacing may be specified, which are- passed verbatim to the command without expansion. A `%' as- `option' expands into the pathname, which is the conventional- dash `-' when searching standard input. Option --label=.ext may- be used to force filtering standard input. For example, --fil-- ter='pdf:pdftotext % -' searches PDF files.+ more `option' separated by spacing may be specified, which are+ passed verbatim to the command. A `%' as `option' expands into+ the pathname to search. For example, --filter='pdf:pdftotext %+ -' searches PDF files. The `%' expands into a `-' when search-+ ing standard input. Option --label=.ext may be used to specify+ extension `ext' when searching standard input. --format=FORMAT Output FORMAT-formatted matches. See `man ugrep' section FORMAT- for the `%' fields. Options -A, -B, -C, -y, and -v are dis-+ for the `%' fields. Options -A, -B, -C, -y, and -v are dis- abled. --free-space@@ -2507,54 +2577,57 @@ in markdown: behave as traditional grep. -g GLOB, --glob=GLOB- Search only files whose name matches GLOB, same as- --include=GLOB. If GLOB is preceded by a `!', skip files whose- name matches GLOB, same as --exclude=GLOB.+ Search only files whose name matches GLOB, same as+ --include=GLOB. When GLOB is preceded by a `!' or a `^', skip+ files whose name matches GLOB, same as --exclude=GLOB. --group-separator[=SEP] Use SEP as a group separator for context options -A, -B, and -C. The default is a double hyphen (`--'). -H, --with-filename- Always print the filename with output lines. This is the+ Always print the filename with output lines. This is the default when there is more than one file to search. -h, --no-filename- Never print filenames with output lines. This is the default- when there is only one file (or only standard input) to search.+ Never print filenames with output lines. This is the default+ when there is only one file (or only standard input) to search. --help Print a help message. - -I Ignore matches in binary files. This option is equivalent to+ -I Ignore matches in binary files. This option is equivalent to the --binary-files=without-match option. -i, --ignore-case- Perform case insensitive matching. By default, ugrep is case+ Perform case insensitive matching. By default, ugrep is case sensitive. This option applies to ASCII letters only. --ignore-files[=FILE]- Ignore files and directories specified in a FILE when encoun-- tered in recursive searches. The default is `.gitignore'.- Files and directories matching the globs in FILE are ignored in- the directory tree rooted at each FILE's location by temporarily- overriding --exclude and --exclude-dir globs. This option may- be repeated.+ Ignore files and directories matching the globs in each FILE+ when encountered in recursive searches. The default FILE is+ `.gitignore'. Matching files and directories located in the+ directory tree rooted at a FILE's location are ignored by tempo-+ rarily overriding the --exclude and --exclude-dir globs. Files+ and directories specified as FILE arguments are not ignored.+ This option may be repeated. --include=GLOB Search only files whose name matches GLOB using wildcard match- ing, same as -g GLOB. GLOB can use **, *, ?, and [...] as wild- cards, and \ to quote a wildcard or backslash character liter-- ally. If GLOB contains /, file pathnames are matched. Other-- wise file basenames are matched. Note that --exclude patterns+ ally. When GLOB contains a `/', full pathnames are matched.+ Otherwise basenames are matched. Note that --exclude patterns take priority over --include patterns. This option may be repeated. --include-dir=GLOB Only directories whose name matches GLOB are included in recur-- sive searches. If GLOB contains /, full pathnames are matched.- Otherwise basenames are matched. Note that --exclude-dir pat-- terns take priority over --include-dir patterns. This option- may be repeated.+ sive searches. GLOB can use **, *, ?, and [...] as wildcards,+ and \ to quote a wildcard or backslash character literally.+ When GLOB contains a `/', full pathnames are matched. Otherwise+ basenames are matched. Note that --exclude-dir patterns take+ priority over --include-dir patterns. This option may be+ repeated. --include-from=FILE Read the globs from FILE and search only files and directories@@ -2564,12 +2637,12 @@ in markdown: input is read. This option may be repeated. --include-fs=MOUNTS- Only file systems with mounting points MOUNTS are included in- recursive searches. MOUNTS is a comma-separated list of path-- names. --include-fs=. restricts recursive searches to the file- system of the working directory only. Note that --exclude-fs- mounts take priority over --include-fs mounts. This option may- be repeated.+ Only file systems specified by MOUNTS are included in recursive+ searches. MOUNTS is a comma-separated list of mount points or+ pathnames of directories on file systems. --include-fs=.+ restricts recursive searches to the file system of the working+ directory only. Note that --exclude-fs mounts take priority+ over --include-fs mounts. This option may be repeated. -J NUM, --jobs=NUM Specifies the number of threads spawned to search files. By@@ -2583,8 +2656,8 @@ in markdown: only. --json Output file matches in JSON. When option -H, -n, -k, or -b is- used additional JSON properties are output. See also option- --format.+ used additional values are output. See also the --format and -u+ options. -K NUM1[,NUM2], --range=NUM1[,NUM2] Start searching at line NUM1 and end at line NUM2 when speci-@@ -2593,7 +2666,7 @@ in markdown: -k, --column-number The column number of a matched pattern is displayed in front of the respective matched line, starting at column 1. Tabs are- expanded when columns are counted, see option --tabs.+ expanded when columns are counted, see also option --tabs. -L, --files-without-match Only the names of files not containing selected lines are writ-@@ -2621,63 +2694,66 @@ in markdown: -M MAGIC, --file-magic=MAGIC Only files matching the signature pattern MAGIC are searched. The signature "magic bytes" at the start of a file are compared- to the `MAGIC' regex pattern. When matching, the file will be- searched. This option may be repeated and may be combined with- options -O and -t to expand the search. This option is rela-- tively slow as every file on the search path is read to compare- MAGIC.+ to the MAGIC regex pattern. When matching, the file will be+ searched. When MAGIC is preceded by a `!' or a `^', skip files+ with matching MAGIC signatures. This option may be repeated and+ may be combined with options -O and -t to expand the search.+ Every file on the search path is read, making searches poten-+ tially more expensive. -m NUM, --max-count=NUM- Stop reading the input after NUM matches for each file pro-+ Stop reading the input after NUM matches for each file pro- cessed. --match Match all input. Same as specifying an empty pattern to search. --max-depth=NUM- Restrict recursive search to NUM (NUM > 0) directories deep,+ Restrict recursive search to NUM (NUM > 0) directories deep, where --max-depth=1 searches the specified path without visiting- sub-directories. By comparison, -dskip skips all directories+ sub-directories. By comparison, -dskip skips all directories even when they are on the command line. --max-files=NUM- If -R or -r is specified, restrict the number of files matched- to NUM. Specify -J1 to produce replicable results by ensuring+ If -R or -r is specified, restrict the number of files matched+ to NUM. Specify -J1 to produce replicable results by ensuring that files are searched in the same order as specified. -N PATTERN, --neg-regexp=PATTERN- Specify a negative PATTERN used during the search of the input:- an input line is selected only if it matches any of the speci-- fied patterns when PATTERN does not match. Same as -e (?^PAT-- TERN) Negative PATTERN matches are removed before any other- specified patterns are matched. Note that longer patterns take- precedence over shorter patterns. This option may be repeated.+ Specify a negative PATTERN used during the search of the input:+ an input line is selected only if it matches any of the speci-+ fied patterns when PATTERN does not match. Same as -e (?^PAT-+ TERN). Negative PATTERN matches are removed before any other+ specified patterns are matched. Note that longer patterns take+ precedence over shorter patterns. This option may be repeated. -n, --line-number- Each output line is preceded by its relative line number in the- file, starting at line 1. The line number counter is reset for+ Each output line is preceded by its relative line number in the+ file, starting at line 1. The line number counter is reset for each file processed. --no-group-separator- Removes the group separator line from the output for context+ Removes the group separator line from the output for context options -A, -B, and -C. --[no-]hidden Do (not) search hidden files and directories. --[no-]mmap- Do (not) use memory maps to search files. By default, memory+ Do (not) use memory maps to search files. By default, memory maps are used under certain conditions to improve performance. -O EXTENSIONS, --file-extensions=EXTENSIONS- Search only files whose filename extensions match the specified- comma-separated list of EXTENSIONS. This option is the same as- specifying --include='*.ext' for each extension suffix `ext' in- the EXTENSIONS list. This option may be repeated and may be- combined with options -M and -t to expand the recursive search.+ Search only files whose filename extensions match the specified+ comma-separated list of EXTENSIONS, same as --include='*.ext'+ for each `ext' in EXTENSIONS. When `ext' is preceded by a `!'+ or a `^', skip files whose filename extensions matches `ext',+ same as --exclude='*.ext'. This option may be repeated and may+ be combined with options -M and -t to expand the recursive+ search. -o, --only-matching- Prints only the matching part of lines. When multiple lines+ Print only the matching part of lines. When multiple lines match, the line numbers with option -n are displayed using `|' as the field separator for each additional line matched by the pattern. This option cannot be combined with options -A, -B,@@ -2720,12 +2796,12 @@ in markdown: -R, --dereference-recursive Recursively read all files under each directory. Follow all- symbolic links, unlike -r. If -J1 is specified, files are+ symbolic links, unlike -r. When -J1 is specified, files are searched in the same order as specified. -r, --recursive Recursively read all files under each directory, following sym-- bolic links only if they are on the command line. If -J1 is+ bolic links only if they are on the command line. When -J1 is specified, files are searched in the same order as specified. -S, --dereference@@ -2743,8 +2819,8 @@ in markdown: --stats Display statistics on the number of files and directories- searched. Display MAGIC and GLOB selections applied to recur-- sive searches.+ searched. Display the inclusion and exclusion constraints+ applied. -T, --initial-tab Add a tab space to separate the file name, line number, column@@ -2752,26 +2828,26 @@ in markdown: -t TYPES, --file-type=TYPES Search only files associated with TYPES, a comma-separated list- of file types. Each file type corresponds to a set of file name+ of file types. Each file type corresponds to a set of filename extensions passed to option -O. For capitalized file types, the- search is expanded to include files found on the search path- with matching file signature magic bytes passed to option -M.- This option may be repeated. The possible values of TYPES can- be (use option -tlist to display a more detailed list): `action-- script', `ada', `asm', `asp', `aspx', `autoconf', `automake',- `awk', `Awk', `basic', `batch', `bison', `c', `c++', `clojure',- `csharp', `css', `csv', `dart', `Dart', `delphi', `elisp',- `elixir', `erlang', `fortran', `gif', `Gif', `go', `groovy',- `gsp', `haskell', `html', `jade', `java', `jpeg', `Jpeg', `js',- `json', `jsp', `julia', `kotlin', `less', `lex', `lisp', `lua',- `m4', `make', `markdown', `matlab', `node', `Node', `objc',- `objc++', `ocaml', `parrot', `pascal', `pdf', `Pdf', `perl',- `Perl', `php', `Php', `png', `Png', `prolog', `python',- `Python', `r', `rpm', `Rpm', `rst', `rtf', `Rtf', `ruby',- `Ruby', `rust', `scala', `scheme', `shell', `Shell',- `smalltalk', `sql', `svg', `swift', `tcl', `tex', `text',- `tiff', `Tiff', `tt', `typescript', `verilog', `vhdl', `vim',- `xml', `Xml', `yacc', `yaml'.+ search is expanded to include files with matching file signature+ magic bytes, as if passed to option -M. When a type is pre-+ ceeded by a `!' or a `^', excludes files of the specified type.+ This option may be repeated. The possible file types can be+ (where -tlist displays a detailed list): `actionscript', `ada',+ `asm', `asp', `aspx', `autoconf', `automake', `awk', `Awk',+ `basic', `batch', `bison', `c', `c++', `clojure', `csharp',+ `css', `csv', `dart', `Dart', `delphi', `elisp', `elixir',+ `erlang', `fortran', `gif', `Gif', `go', `groovy', `gsp',+ `haskell', `html', `jade', `java', `jpeg', `Jpeg', `js', `json',+ `jsp', `julia', `kotlin', `less', `lex', `lisp', `lua', `m4',+ `make', `markdown', `matlab', `node', `Node', `objc', `objc++',+ `ocaml', `parrot', `pascal', `pdf', `Pdf', `perl', `Perl',+ `php', `Php', `png', `Png', `prolog', `python', `Python', `r',+ `rpm', `Rpm', `rst', `rtf', `Rtf', `ruby', `Ruby', `rust',+ `scala', `scheme', `shell', `Shell', `smalltalk', `sql', `svg',+ `swift', `tcl', `tex', `text', `tiff', `Tiff', `tt', `type-+ script', `verilog', `vhdl', `vim', `xml', `Xml', `yacc', `yaml'. --tabs=NUM Set the tab size to NUM to expand tabs for option -k. The value@@ -2779,12 +2855,12 @@ in markdown: -U, --binary Disables Unicode matching for binary file matching, forcing PAT-- TERN to match bytes, not Unicode characters. For example, -U- '\xa3' matches byte A3 (hex) instead of the Unicode code point+ TERN to match bytes, not Unicode characters. For example, -U+ '\xa3' matches byte A3 (hex) instead of the Unicode code point U+00A3 represented by the two-byte UTF-8 sequence C2 A3. -u, --ungroup- Do not group multiple pattern matches on the same matched line.+ Do not group multiple pattern matches on the same matched line. Output the matched line again for each additional pattern match, using `+' as the field separator. @@ -2792,82 +2868,80 @@ in markdown: Display version information and exit. -v, --invert-match- Selected lines are those not matching any of the specified pat-+ Selected lines are those not matching any of the specified pat- terns. -W, --with-hex- Output binary matches in hexadecimal, leaving text matches+ Output binary matches in hexadecimal, leaving text matches alone. This option is equivalent to the --binary-files=with-hex option. -w, --word-regexp- The PATTERN or -e PATTERN is searched for as a word (as if sur-- rounded by \< and \>). If PATTERN or -e PATTERN is also speci-- fied, then this option does not apply to -f FILE patterns.+ The PATTERN is searched for as a word (as if surrounded by \<+ and \>). If a PATTERN is specified (or -e PATTERN or -N PAT-+ TERN), then this option does not apply to -f FILE patterns. -X, --hex Output matches in hexadecimal. This option is equivalent to the --binary-files=hex option. -x, --line-regexp- Only input lines selected against the entire PATTERN or -e PAT-- TERN is considered to be matching lines (as if surrounded by ^- and $). If PATTERN or -e PATTERN is also specified, then this+ Only input lines selected against the entire PATTERN is consid-+ ered to be matching lines (as if surrounded by ^ and $). If a+ PATTERN is specified (or -e PATTERN or -N PATTERN), then this option does not apply to -f FILE patterns. - --xml Output file matches in XML. When option -H, -n, -k, or -b is- used additional XML attributes are output. See also option- --format.+ --xml Output file matches in XML. When option -H, -n, -k, or -b is+ used additional values are output. See also the --format and -u+ options. -Y, --empty- Permits empty matches, such as `^\h*$' to match blank lines.- Empty matches are disabled by default. Note that empty-matching- patterns, such as x? and x*, match all input with -Y, not only- lines with `x'.+ Permits empty matches. By default, empty matches are disabled,+ unless a pattern starts with `^' and ends with `$'. Note that+ -Y when specified with an empty-matching pattern such as x? and+ x*, match all input, not only lines with a `x'. -y, --any-line Any matching or non-matching line is output. Non-matching lines- are output with the `-' separator as context of the matching+ are output with the `-' separator as context of the matching lines. See also the -A, -B, and -C options. -Z, --null Prints a zero-byte after the file name. -z, --decompress- Decompress files to search, when compressed. Archives (.cpio,- .pax, .tar, .zip) and compressed archives (e.g. .taz, .tgz,- .tpz, .tbz, .tbz2, .tb2, .tz2, .tlz, and .txz) are searched and- matching pathnames of files in archives are output in braces.- If -g, -O, -M, or -t is specified, searches files within ar-- chives whose name matches globs, matches file name extensions,- matches file signature magic bytes, or matches file types,- respectively. Option --label=.ext associates suffix `ext' with- standard input. Supported compression formats: gzip (default,- optional suffix .gz), compress (requires suffix .Z), zip- (requires suffix .zip or .ZIP), bzip2 (requires suffix .bz,- .bz2, or .bzip2), lzma (requires suffix .lzma), and xz (requires- suffix .xz).+ Decompress files to search, when compressed. Archives (.cpio,+ .jar, .pax, .tar, .zip) and compressed archives (e.g. .taz,+ .tgz, .tpz, .tbz, .tbz2, .tb2, .tz2, .tlz, and .txz) are+ searched and matching pathnames of files in archives are output+ in braces. If -g, -O, -M, or -t is specified, searches files+ within archives whose name matches globs, matches file name+ extensions, matches file signature magic bytes, or matches file+ types, respectively. Supported compression formats: gzip (.gz),+ compress (.Z), zip, bzip2 (requires suffix .bz, .bz2, .bzip2,+ .tbz, .tbz2, .tb2, .tz2), lzma and xz (requires suffix .lzma,+ .tlz, .xz, .txz). If no FILE arguments are specified, or if a `-' is specified, the stan- dard input is used, unless recursive searches are specified which exam- ine the working directory. - If no FILE arguments are specified and one of the options -g, -O, -M,+ If no FILE arguments are specified and one of the options -g, -O, -M, -t, --include, --include-dir, --exclude, or --exclude-dir is specified, recursive searches are performed as if -r was specified. - A `--' signals the end of options; the rest of the parameters are FILE+ A `--' signals the end of options; the rest of the parameters are FILE arguments, allowing filenames to begin with a `-' character. - The regular expression pattern syntax is an extended form of the POSIX+ The regular expression pattern syntax is an extended form of the POSIX ERE syntax. For an overview of the syntax see README.md or visit: https://github.com/Genivia/ugrep - Note that `.' matches any non-newline character. Pattern `\n' matches- a newline character. Multiple lines may be matched with patterns that- match newlines, unless one or more of the context options -A, -B,- -C or -y is used, or option -v.+ Note that `.' matches any non-newline character. Pattern `\n' matches+ a newline character. Multiple lines may be matched with patterns that+ match newlines, unless one or more of the context options -A, -B, -C,+ or -y is used, or option -v is used. EXIT STATUS The ugrep utility exits with one of the following values:@@ -2889,8 +2963,11 @@ in markdown: arator `/', the pathname is matched. Otherwise the basename of a file or directory is matched. For example, *.h matches foo.h and bar/foo.h. bar/*.h matches bar/foo.h but not foo.h and not bar/bar/foo.h. Use a- leading `/' to force /*.h to match foo.h but not bar/foo.h. A glob- starting with a `!' is negated, i.e. does not match.+ leading `/' to force /*.h to match foo.h but not bar/foo.h.++ When a glob starts with a `!' as specified with -g!GLOB, or specified+ in a FILE with --include-from=FILE or --exclude-from=FILE, it is+ negated. Glob Syntax and Conventions @@ -2904,7 +2981,7 @@ in markdown: [!a-z] Matches one character not in the selected range of characters. - / When used at the begin of a glob, matches if pathname has no /.+ / When used at the begin of a glob, matches if pathname has no /. **/ Matches zero or more directories. @@ -2945,34 +3022,34 @@ in markdown: a\?b Matches a?b, but not a, b, ab, axb, a/b - Lines in the --exclude-from and --include-from files are ignored when- empty or start with a `#'. The prefix `!' to a glob in such a file- negates the pattern match, i.e. matching files are excluded except- files matching the globs prefixed with `!' in the --exclude-from file.+ Lines in the --exclude-from and --include-from files are ignored when+ empty or start with a `#'. The prefix `!' to a glob in such a file+ negates the pattern match, i.e. matching files are excluded except+ files matching the globs prefixed with `!' in the --exclude-from file. ENVIRONMENT GREP_PATH- May be used to specify a file path to pattern files. The file- path is used by option -f to open a pattern file, when the file+ May be used to specify a file path to pattern files. The file+ path is used by option -f to open a pattern file, when the file cannot be opened. GREP_COLOR- May be used to specify ANSI SGR parameters to highlight matches- when option --color is used, e.g. 1;35;40 shows pattern matches+ May be used to specify ANSI SGR parameters to highlight matches+ when option --color is used, e.g. 1;35;40 shows pattern matches in bold magenta text on a black background. GREP_COLORS- May be used to specify ANSI SGR parameters to highlight matches- and other attributes when option --color is used. Its value is- a colon-separated list of ANSI SGR parameters that defaults to+ May be used to specify ANSI SGR parameters to highlight matches+ and other attributes when option --color is used. Its value is+ a colon-separated list of ANSI SGR parameters that defaults to cx=33:mt=1;31:fn=1;35:ln=1;32:cn=1;32:bn=1;32:se=36. The mt=,- ms=, and mc= capabilities of GREP_COLORS have priority over+ ms=, and mc= capabilities of GREP_COLORS have priority over GREP_COLOR. Option --colors has priority over GREP_COLORS. GREP_COLORS- Colors are specified as a substring of semicolon-separated ANSI SGR- codes to highlight specific parts of the output when option- --color=auto or --color=always is used:+ Colors are specified as string of colon-separated ANSI SGR parameters+ of the form `what=substring', where `substring' is a semicolon-sepa-+ rated list of SGR codes to color-highlight `what': sl= SGR substring for selected lines. @@ -2982,10 +3059,10 @@ in markdown: mt= SGR substring for matching text in any matching line. - ms= SGR substring for matching text in a selected line. The sub-+ ms= SGR substring for matching text in a selected line. The sub- string mt= by default. - mc= SGR substring for matching text in a context line. The sub-+ mc= SGR substring for matching text in a context line. The sub- string mt= by default. fn= SGR substring for file names.@@ -2999,7 +3076,7 @@ in markdown: se= SGR substring for separators. FORMAT- Option --format=FORMAT specifies an output format for file matches.+ Option --format=FORMAT specifies an output format for file matches. Fields may be used in FORMAT, which expand into the following values: %[ARG]F@@ -3094,28 +3171,28 @@ in markdown: %% the percentage sign. - %1 the first regex group capture of the match, and so on up to- group %9, same as %[1]#, requires option -P Perl matching.+ %1 the first regex group capture of the match, and so on up to+ group %9, same as %[1]#; requires option -P Perl matching. %[NUM]#- the regex group capture NUM, requires option -P.+ the regex group capture NUM; requires option -P Perl matching. - The [ARG] part of a field is optional and may be omitted. When- present, the argument must be placed in [] brackets, for example %[,]F+ The [ARG] part of a field is optional and may be omitted. When+ present, the argument must be placed in [] brackets, for example %[,]F to output a comma, the pathname, and a separator. - Fields %[SEP]$ and %u are switches and do not write anything to the- output.+ Fields %[SEP]$ and %u are switches and do not send anything to the out-+ put. The separator used by %P, %H, %N, %K, %B, and %S may be changed by pre-- ceeding the field with a %[SEP]$. When [SEP] is not provided, reverses- the separator to the default separator or the separator specified by- --separator.+ ceeding the field by %[SEP]$. When [SEP] is not provided, this+ reverses the separator to the default separator or the separator speci-+ fied with --separator. Formatted output is written for each matching pattern, which means that- a line may be output multiple times when patterns match more than once- on the same line. When field %u is found anywhere in the specified- format string, matching lines are output only once unless option -u,+ a line may be output multiple times when patterns match more than once+ on the same line. When field %u is found anywhere in the specified+ format string, matching lines are output only once unless option -u, --ungroup is used or when a newline is matched. Additional formatting options:@@ -3132,7 +3209,7 @@ in markdown: --format-end=FORMAT the FORMAT when ending the search. - The context options -A, -B, -C, -y, and options -v, --break, --color,+ The context options -A, -B, -C, -y, and options -v, --break, --color, -T, and --null are disabled and have no effect on the formatted output. EXAMPLES@@ -3177,7 +3254,7 @@ in markdown: $ ugrep -n -f c++/comments myfile.cpp - List the lines that need fixing in a C/C++ source file by looking for+ List the lines that need fixing in a C/C++ source file by looking for the word `FIXME' while skipping any `FIXME' in quoted strings: $ ugrep -e 'FIXME' -N '"(\\.|\\\r?\n|[^\\\n"])*"' myfile.cpp@@ -3211,14 +3288,14 @@ in markdown: $ ugrep -z -tc++ -n 'FIXME' project.tgz - Recursively search for the word `copyright' in cpio/pax/tar/zip ar-+ Recursively search for the word `copyright' in cpio/jar/pax/tar/zip ar- chives, compressed and regular files, and in PDFs using a PDF filter: $ ugrep -r -z -w --filter='pdf:pdftotext % -' 'copyright' - Match the binary pattern `A3hhhhA3hh' (hex) in a binary file without- Unicode pattern matching -U (which would otherwise match `\xaf' as a- Unicode character U+00A3 with UTF-8 byte sequence C2 A3) and display+ Match the binary pattern `A3hhhhA3hh' (hex) in a binary file without+ Unicode pattern matching -U (which would otherwise match `\xaf' as a+ Unicode character U+00A3 with UTF-8 byte sequence C2 A3) and display the results in hex with -X using `less -R' as a pager: $ ugrep --pager -UXo '\xa3[\x00-\xff]{2}\xa3[\x00-\xff]' a.out@@ -3231,7 +3308,7 @@ in markdown: $ ugrep -Rl '' --ignore-files - List all files containing a RPM signature, located in the `rpm' direc-+ List all files containing a RPM signature, located in the `rpm' direc- tory and recursively below up to two levels deeper: $ ugrep -R --max-depth=3 -l -tRpm '' rpm/@@ -3247,8 +3324,8 @@ in markdown: LICENSE- ugrep is released under the BSD-3 license. All parts of the software- have reasonable copyright terms permitting free redistribution. This+ ugrep is released under the BSD-3 license. All parts of the software+ have reasonable copyright terms permitting free redistribution. This includes the ability to reuse all or parts of the ugrep source tree. SEE ALSO@@ -3256,7 +3333,7 @@ in markdown: - ugrep 1.7.2 January 12, 2020 UGREP(1)+ ugrep 1.7.3 January 20, 2020 UGREP(1) <a name="patterns"/>
bin/linux/ugrep
bin/macosx/ugrep
@@ -856,11 +856,11 @@ class AbstractMatcher { (void)lineno(); return bol_; }- /// Returns pointer to the end of line in the buffer after the matched text, DANGER: invalidates previous bol() and text() pointers, only use eol() before bol() and text().+ /// Returns pointer to the end of line in the buffer after the matched text, DANGER: invalidates previous bol() and text() pointers, use eol() before bol(), text(), begin(), and end() when those are required. inline const char *eol(bool inclusive = false) ///< true if eol should be inclusive, i.e. after \n /// @returns pointer to the end of line {- if (chr_ == '\n' || txt_[len_] == '\n')+ if (chr_ == '\n' || (txt_ + len_ < buf_ + end_ && txt_[len_] == '\n')) return txt_ + len_ + inclusive; size_t tmp = pos_; while (true)@@ -876,7 +876,9 @@ class AbstractMatcher { } if (eof_) break;+ pos_ = tmp; (void)grow();+ tmp = pos_; pos_ = end_; end_ += get(buf_ + end_, blk_ ? blk_ : max_ - end_ - 1); if (pos_ >= end_ && !wrap())@@ -1161,7 +1163,7 @@ class AbstractMatcher { virtual size_t match(Method method) /// @returns nonzero when input matched the pattern using method Const::SCAN, Const::FIND, Const::SPLIT, or Const::MATCH = 0;- /// Shift or expand the internal buffer when it is too small to accommodate more input, where the buffer size is doubled when needed.+ /// Shift or expand the internal buffer when it is too small to accommodate more input, where the buffer size is doubled when needed, change cur_, pos_, end_, max_, ind_, buf_, bol_, lpb_, and txt_. bool grow(size_t need = Const::BLOCK) ///< optional needed space = Const::BLOCK size by default /// @returns true if buffer was shifted or was enlarged {
@@ -431,7 +431,7 @@ class Input { { init(); }- /// Construct input character sequence from an open FILE* file descriptor, supports UTF-8 conversion from UTF-16 and UTF-32.+ /// Construct input character sequence from an open FILE* file descriptor, using the specified file encoding Input( FILE *file, ///< input file file_encoding_type enc, ///< file_encoding (when UTF BOM is not present)@@ -928,6 +928,17 @@ class BufferedInput : public Input { std::memcpy(buf_, input.buf_, len_); return *this; }+ /// Construct buffered input character sequence from an open FILE* file descriptor, using the specified file encoding+ BufferedInput(+ FILE *file, ///< input file+ file_encoding_type enc, ///< file_encoding (when UTF BOM is not present)+ const unsigned short *page = NULL) ///< code page for file_encoding::custom+ :+ Input(file, enc, page)+ {+ len_ = Input::get(buf_, SIZE);+ pos_ = 0;+ } // Cast this Input object to bool, same as checking good(). operator bool() /// @returns true if a non-empty sequence of characters is available to get
@@ -265,17 +265,23 @@ class Pattern { { return assign(fsm); }- /// Number of subpatterns of this pattern object.+ /// Get the number of subpatterns of this pattern object. Index size() const /// @returns number of subpatterns { return static_cast<Index>(end_.size()); }+ /// Return true if this pattern is not assigned.+ bool empty() const+ /// @return true if this pattern is not assigned+ {+ return opc_ == NULL && fsm_ == NULL;+ } /// Get subpattern regex of this pattern object or the whole regex with index 0. const std::string operator[](Index choice) const /// @returns subpattern string or "" when not set ;- /// Check is subpattern is reachable by a match.+ /// Check if subpattern is reachable by a match. bool reachable(Index choice) const /// @returns true if subpattern is reachable {
@@ -83,7 +83,7 @@ https://github.com/Genivia/ugrep Note that `.' matches any non-newline character. Pattern `\\n' matches a newline character. Multiple lines may be matched with patterns that match newlines, unless one or more of the context options \fB-A\fR, \fB-B\fR,- \fB-C\fR or \fB-y\fR is used, or option \fB-v\fR.+\fB-C\fR, or \fB-y\fR is used, or option \fB-v\fR is used. .SH "EXIT STATUS" The \fBugrep\fR utility exits with one of the following values: .IP 0@@ -104,8 +104,11 @@ a glob contains a path separator `/', the pathname is matched. Otherwise the basename of a file or directory is matched. For example, \fB*.h\fR matches \fIfoo.h\fR and \fIbar/foo.h\fR. \fBbar/*.h\fR matches \fIbar/foo.h\fR but not \fIfoo.h\fR and not \fIbar/bar/foo.h\fR. Use a leading `/' to force \fB/*.h\fR-to match \fIfoo.h\fR but not \fIbar/foo.h\fR. A glob starting with a `!' is-negated, i.e. does not match.+to match \fIfoo.h\fR but not \fIbar/foo.h\fR.+.PP+When a glob starts with a `!' as specified with \fB-g\fR!\fIGLOB\fR, or+specified in a \fIFILE\fR with \fB--include-from\fR=\fIFILE\fR or+\fB--exclude-from\fR=\fIFILE\fR, it is negated. .PP \fBGlob Syntax and Conventions\fR .IP \fB*\fR@@ -177,9 +180,9 @@ list of ANSI SGR parameters that defaults to \fBms=\fR, and \fBmc=\fR capabilities of \fBGREP_COLORS\fR have priority over \fBGREP_COLOR\fR. Option \fB--colors\fR has priority over \fBGREP_COLORS\fR. .SH GREP_COLORS-Colors are specified as a substring of semicolon-separated ANSI SGR codes to-highlight specific parts of the output when option \fB--color=auto\fR or-\fB--color=always\fR is used:+Colors are specified as string of colon-separated ANSI SGR parameters of the+form `what=substring', where `substring' is a semicolon-separated list of SGR+codes to color-highlight `what': .IP \fBsl=\fR SGR substring for selected lines. .IP \fBcx=\fR@@ -291,22 +294,21 @@ if not the first match: a verical bar, same as \fB%[|]>\fR. the percentage sign. .IP \fB%1\fR the first regex group capture of the match, and so on up to group \fB%9\fR,-same as \fB%[1]#\fR, requires option \fB-P\fR Perl matching.+same as \fB%[1]#\fR; requires option \fB-P\fR Perl matching. .IP \fB%[\fINUM\fR\fB]#\fR-the regex group capture \fINUM\fR, requires option \fB-P\fR.+the regex group capture \fINUM\fR; requires option \fB-P\fR Perl matching. .PP The \fB[\fR\fIARG\fR\fB]\fR part of a field is optional and may be omitted. When present, the argument must be placed in \fB[]\fR brackets, for example \fB%[,]F\fR to output a comma, the pathname, and a separator. .PP-Fields \fB%[\fR\fISEP\fR\fB]$\fR and \fB%u\fR are switches and do not write+Fields \fB%[\fR\fISEP\fR\fB]$\fR and \fB%u\fR are switches and do not send anything to the output. .PP The separator used by \fB%P\fR, \fB%H\fR, \fB%N\fR, \fB%K\fR, \fB%B\fR, and-\fB%S\fR may be changed by preceeding the field with a-\fB%[\fR\fISEP\fR\fB]$\fR. When \fB[\fR\fISEP\fR\fB]\fR is not provided,-reverses the separator to the default separator or the separator specified by-\fB--separator\fR.+\fB%S\fR may be changed by preceeding the field by \fB%[\fR\fISEP\fR\fB]$\fR.+When \fB[\fR\fISEP\fR\fB]\fR is not provided, this reverses the separator to+the default separator or the separator specified with \fB--separator\fR. .PP Formatted output is written for each matching pattern, which means that a line may be output multiple times when patterns match more than once on the same@@ -403,7 +405,7 @@ Find lines with `FIXME' in the C/C++ files stored in a tarball: .IP $ ugrep -z -tc++ -n 'FIXME' project.tgz .PP-Recursively search for the word `copyright' in cpio/pax/tar/zip archives,+Recursively search for the word `copyright' in cpio/jar/pax/tar/zip archives, compressed and regular files, and in PDFs using a PDF filter: .IP $ ugrep -r -z -w --filter='pdf:pdftotext % -' 'copyright'
@@ -1,4 +1,4 @@-.TH UGREP "1" "January 12, 2020" "ugrep 1.7.2" "User Commands"+.TH UGREP "1" "January 20, 2020" "ugrep 1.7.3" "User Commands" .SH NAME \fBugrep\fR -- universal file pattern searcher .SH SYNOPSIS@@ -61,8 +61,8 @@ zero byte or invalid UTF encoding. See also the \fB\-a\fR, \fB\-I\fR, \fB\-U\fR and \fB\-X\fR options. .TP \fB\-\-break\fR-Groups matches per file and adds a line break between results from-different files.+Group matches per file. Adds a header and line break between+results from different files. .TP \fB\-C\fR[NUM], \fB\-\-context\fR[=\fINUM\fR] Print NUM lines of leading and trailing context surrounding each@@ -88,11 +88,11 @@ list of ANSI SGR parameters. COLORS selectively overrides the colors specified by the GREP_COLORS environment variable. .TP \fB\-\-cpp\fR-Output file matches in C++. See also option \fB\-\-format\fR.+Output file matches in C++. See also the \fB\-\-format\fR and \fB\-u\fR options. .TP \fB\-\-csv\fR Output file matches in CSV. When option \fB\-H\fR, \fB\-n\fR, \fB\-k\fR, or \fB\-b\fR is used-additional CSV values are output. See also option \fB\-\-format\fR.+additional values are output. See also the \fB\-\-format\fR and \fB\-u\fR options. .TP \fB\-D\fR \fIACTION\fR, \fB\-\-devices\fR=\fIACTION\fR If an input file is a device, FIFO or socket, use ACTION to process@@ -126,17 +126,18 @@ to specify a pattern after option \fB\-f\fR or after the FILE arguments. \fB\-\-exclude\fR=\fIGLOB\fR Skip files whose name matches GLOB using wildcard matching, same as \fB\-g\fR \fI!GLOB\fR. GLOB can use **, *, ?, and [...] as wildcards, and \\ to-quote a wildcard or backslash character literally. If GLOB-contains /, full pathnames are matched. Otherwise basenames are-matched. Note that \fB\-\-exclude\fR patterns take priority over \fB\-\-include\fR-patterns. This option may be repeated.+quote a wildcard or backslash character literally. When GLOB+contains a `/', full pathnames are matched. Otherwise basenames+are matched. Note that \fB\-\-exclude\fR patterns take priority over+\fB\-\-include\fR patterns. This option may be repeated. .TP \fB\-\-exclude\-dir\fR=\fIGLOB\fR Exclude directories whose name matches GLOB from recursive-searches. If GLOB contains /, full pathnames are matched.-Otherwise basenames are matched. Note that \fB\-\-exclude\-dir\fR patterns-take priority over \fB\-\-include\-dir\fR patterns. This option may be-repeated.+searches. GLOB can use **, *, ?, and [...] as wildcards, and \\ to+quote a wildcard or backslash character literally. When GLOB+contains a `/', full pathnames are matched. Otherwise basenames+are matched. Note that \fB\-\-exclude\-dir\fR patterns take priority over+\fB\-\-include\-dir\fR patterns. This option may be repeated. .TP \fB\-\-exclude\-from\fR=\fIFILE\fR Read the globs from FILE and skip files and directories whose name@@ -146,10 +147,10 @@ are ignored. When FILE is a `\fB\-'\fR, standard input is read. This option may be repeated. .TP \fB\-\-exclude\-fs\fR=\fIMOUNTS\fR-Exclude file systems with mounting points MOUNTS from recursive-searches, MOUNTS is a comma\-separated list of pathnames. Note-that \fB\-\-exclude\-fs\fR mounts take priority over \fB\-\-include\-fs\fR mounts.-This option may be repeated.+Exclude file systems specified by MOUNTS from recursive searches,+MOUNTS is a comma\-separated list of mount points or pathnames of+directories on file systems. Note that \fB\-\-exclude\-fs\fR mounts take+priority over \fB\-\-include\-fs\fR mounts. This option may be repeated. .TP \fB\-F\fR, \fB\-\-fixed\-strings\fR Interpret pattern as a set of fixed strings, separated by newlines,@@ -172,11 +173,11 @@ where `exts' is a comma\-separated list of filename extensions and `command' is a filter utility. The filter utility should read from standard input and write to standard output. Files matching one of `exts` are filtered only. One or more `option' separated by-spacing may be specified, which are passed verbatim to the command-without expansion. A `%' as `option' expands into the pathname,-which is the conventional dash `\fB\-'\fR when searching standard input.-Option \fB\-\-label\fR=.ext may be used to force filtering standard input.-For example, \fB\-\-filter\fR='pdf:pdftotext % \fB\-'\fR searches PDF files.+spacing may be specified, which are passed verbatim to the command.+A `%' as `option' expands into the pathname to search. For example,+\fB\-\-filter\fR='pdf:pdftotext % \fB\-'\fR searches PDF files. The `%' expands+into a `\fB\-'\fR when searching standard input. Option \fB\-\-label\fR=.ext may+be used to specify extension `ext' when searching standard input. .TP \fB\-\-format\fR=\fIFORMAT\fR Output FORMAT\-formatted matches. See `man ugrep' section FORMAT@@ -191,8 +192,8 @@ behave as traditional grep. .TP \fB\-g\fR \fIGLOB\fR, \fB\-\-glob\fR=\fIGLOB\fR Search only files whose name matches GLOB, same as \fB\-\-include\fR=\fIGLOB\fR.-If GLOB is preceded by a `!', skip files whose name matches GLOB,-same as \fB\-\-exclude\fR=\fIGLOB\fR.+When GLOB is preceded by a `!' or a `^', skip files whose name+matches GLOB, same as \fB\-\-exclude\fR=\fIGLOB\fR. .TP \fB\-\-group\-separator\fR[=\fISEP\fR] Use SEP as a group separator for context options \fB\-A\fR, \fB\-B\fR, and \fB\-C\fR.@@ -218,26 +219,29 @@ Perform case insensitive matching. By default, ugrep is case sensitive. This option applies to ASCII letters only. .TP \fB\-\-ignore\-files\fR[=\fIFILE\fR]-Ignore files and directories specified in a FILE when encountered-in recursive searches. The default is `.gitignore'. Files and-directories matching the globs in FILE are ignored in the directory-tree rooted at each FILE's location by temporarily overriding-\fB\-\-exclude\fR and \fB\-\-exclude\-dir\fR globs. This option may be repeated.+Ignore files and directories matching the globs in each FILE when+encountered in recursive searches. The default FILE is+`.gitignore'. Matching files and directories located in the+directory tree rooted at a FILE's location are ignored by+temporarily overriding the \fB\-\-exclude\fR and \fB\-\-exclude\-dir\fR globs.+Files and directories specified as FILE arguments are not ignored.+This option may be repeated. .TP \fB\-\-include\fR=\fIGLOB\fR Search only files whose name matches GLOB using wildcard matching, same as \fB\-g\fR \fIGLOB\fR. GLOB can use **, *, ?, and [...] as wildcards,-and \\ to quote a wildcard or backslash character literally. If-GLOB contains /, file pathnames are matched. Otherwise file+and \\ to quote a wildcard or backslash character literally. When+GLOB contains a `/', full pathnames are matched. Otherwise basenames are matched. Note that \fB\-\-exclude\fR patterns take priority over \fB\-\-include\fR patterns. This option may be repeated. .TP \fB\-\-include\-dir\fR=\fIGLOB\fR Only directories whose name matches GLOB are included in recursive-searches. If GLOB contains /, full pathnames are matched.-Otherwise basenames are matched. Note that \fB\-\-exclude\-dir\fR patterns-take priority over \fB\-\-include\-dir\fR patterns. This option may be-repeated.+searches. GLOB can use **, *, ?, and [...] as wildcards, and \\ to+quote a wildcard or backslash character literally. When GLOB+contains a `/', full pathnames are matched. Otherwise basenames+are matched. Note that \fB\-\-exclude\-dir\fR patterns take priority over+\fB\-\-include\-dir\fR patterns. This option may be repeated. .TP \fB\-\-include\-from\fR=\fIFILE\fR Read the globs from FILE and search only files and directories@@ -247,11 +251,12 @@ FILE are ignored. When FILE is a `\fB\-'\fR, standard input is read. This option may be repeated. .TP \fB\-\-include\-fs\fR=\fIMOUNTS\fR-Only file systems with mounting points MOUNTS are included in-recursive searches. MOUNTS is a comma\-separated list of pathnames.-\fB\-\-include\-fs\fR=. restricts recursive searches to the file system of-the working directory only. Note that \fB\-\-exclude\-fs\fR mounts take-priority over \fB\-\-include\-fs\fR mounts. This option may be repeated.+Only file systems specified by MOUNTS are included in recursive+searches. MOUNTS is a comma\-separated list of mount points or+pathnames of directories on file systems. \fB\-\-include\-fs\fR=. restricts+recursive searches to the file system of the working directory+only. Note that \fB\-\-exclude\-fs\fR mounts take priority over+\fB\-\-include\-fs\fR mounts. This option may be repeated. .TP \fB\-J\fR \fINUM\fR, \fB\-\-jobs\fR=\fINUM\fR Specifies the number of threads spawned to search files. By@@ -265,7 +270,7 @@ letter. Case insensitive matching applies to ASCII letters only. .TP \fB\-\-json\fR Output file matches in JSON. When option \fB\-H\fR, \fB\-n\fR, \fB\-k\fR, or \fB\-b\fR is used-additional JSON properties are output. See also option \fB\-\-format\fR.+additional values are output. See also the \fB\-\-format\fR and \fB\-u\fR options. .TP \fB\-K\fR \fINUM\fR1[,NUM2], \fB\-\-range\fR=\fINUM\fR1[,NUM2] Start searching at line NUM1 and end at line NUM2 when specified.@@ -273,7 +278,7 @@ Start searching at line NUM1 and end at line NUM2 when specified. \fB\-k\fR, \fB\-\-column\-number\fR The column number of a matched pattern is displayed in front of the respective matched line, starting at column 1. Tabs are expanded-when columns are counted, see option \fB\-\-tabs\fR.+when columns are counted, see also option \fB\-\-tabs\fR. .TP \fB\-L\fR, \fB\-\-files\-without\-match\fR Only the names of files not containing selected lines are written@@ -301,10 +306,11 @@ otherwise. \fB\-M\fR \fIMAGIC\fR, \fB\-\-file\-magic\fR=\fIMAGIC\fR Only files matching the signature pattern MAGIC are searched. The signature "magic bytes" at the start of a file are compared to-the `MAGIC' regex pattern. When matching, the file will be-searched. This option may be repeated and may be combined with-options \fB\-O\fR and \fB\-t\fR to expand the search. This option is relatively-slow as every file on the search path is read to compare MAGIC.+the MAGIC regex pattern. When matching, the file will be searched.+When MAGIC is preceded by a `!' or a `^', skip files with matching+MAGIC signatures. This option may be repeated and may be combined+with options \fB\-O\fR and \fB\-t\fR to expand the search. Every file on the+search path is read, making searches potentially more expensive. .TP \fB\-m\fR \fINUM\fR, \fB\-\-max\-count\fR=\fINUM\fR Stop reading the input after NUM matches for each file processed.@@ -324,9 +330,9 @@ NUM. Specify \fB\-J\fR1 to produce replicable results by ensuring that files are searched in the same order as specified. .TP \fB\-N\fR \fIPATTERN\fR, \fB\-\-neg\-regexp\fR=\fIPATTERN\fR-Specify a negative PATTERN used during the search of the input: an-input line is selected only if it matches any of the specified-patterns when PATTERN does not match. Same as \fB\-e\fR (?^PATTERN)+Specify a negative PATTERN used during the search of the input:+an input line is selected only if it matches any of the specified+patterns when PATTERN does not match. Same as \fB\-e\fR (?^PATTERN). Negative PATTERN matches are removed before any other specified patterns are matched. Note that longer patterns take precedence over shorter patterns. This option may be repeated.@@ -349,13 +355,14 @@ are used under certain conditions to improve performance. .TP \fB\-O\fR \fIEXTENSIONS\fR, \fB\-\-file\-extensions\fR=\fIEXTENSIONS\fR Search only files whose filename extensions match the specified-comma\-separated list of EXTENSIONS. This option is the same as-specifying \fB\-\-include\fR='*.ext' for each extension suffix `ext' in the-EXTENSIONS list. This option may be repeated and may be combined+comma\-separated list of EXTENSIONS, same as \fB\-\-include\fR='*.ext' for+each `ext' in EXTENSIONS. When `ext' is preceded by a `!' or a+`^', skip files whose filename extensions matches `ext', same as+\fB\-\-exclude\fR='*.ext'. This option may be repeated and may be combined with options \fB\-M\fR and \fB\-t\fR to expand the recursive search. .TP \fB\-o\fR, \fB\-\-only\-matching\fR-Prints only the matching part of lines. When multiple lines match,+Print only the matching part of lines. When multiple lines match, the line numbers with option \fB\-n\fR are displayed using `|' as the field separator for each additional line matched by the pattern. This option cannot be combined with options \fB\-A\fR, \fB\-B\fR, \fB\-C\fR, \fB\-v\fR, and \fB\-y\fR.@@ -398,12 +405,12 @@ match has been found, making searches potentially less expensive. .TP \fB\-R\fR, \fB\-\-dereference\-recursive\fR Recursively read all files under each directory. Follow all-symbolic links, unlike \fB\-r\fR. If \fB\-J\fR1 is specified, files are searched-in the same order as specified.+symbolic links, unlike \fB\-r\fR. When \fB\-J\fR1 is specified, files are+searched in the same order as specified. .TP \fB\-r\fR, \fB\-\-recursive\fR Recursively read all files under each directory, following symbolic-links only if they are on the command line. If \fB\-J\fR1 is specified,+links only if they are on the command line. When \fB\-J\fR1 is specified, files are searched in the same order as specified. .TP \fB\-S\fR, \fB\-\-dereference\fR@@ -421,7 +428,7 @@ number, byte offset, and the matched line. The default is a colon .TP \fB\-\-stats\fR Display statistics on the number of files and directories searched.-Display MAGIC and GLOB selections applied to recursive searches.+Display the inclusion and exclusion constraints applied. .TP \fB\-T\fR, \fB\-\-initial\-tab\fR Add a tab space to separate the file name, line number, column@@ -429,12 +436,13 @@ number, and byte offset with the matched line. .TP \fB\-t\fR \fITYPES\fR, \fB\-\-file\-type\fR=\fITYPES\fR Search only files associated with TYPES, a comma\-separated list of-file types. Each file type corresponds to a set of file name+file types. Each file type corresponds to a set of filename extensions passed to option \fB\-O\fR. For capitalized file types, the-search is expanded to include files found on the search path with-matching file signature magic bytes passed to option \fB\-M\fR. This-option may be repeated. The possible values of TYPES can be-(use option \fB\-t\fRlist to display a more detailed list):+search is expanded to include files with matching file signature+magic bytes, as if passed to option \fB\-M\fR. When a type is preceeded+by a `!' or a `^', excludes files of the specified type. This+option may be repeated. The possible file types can be (where+\fB\-t\fRlist displays a detailed list): `actionscript', `ada', `asm', `asp', `aspx', `autoconf', `automake', `awk', `Awk', `basic', `batch', `bison', `c', `c++', `clojure', `csharp', `css', `csv', `dart', `Dart', `delphi',@@ -477,28 +485,29 @@ Output binary matches in hexadecimal, leaving text matches alone. This option is equivalent to the \fB\-\-binary\-files\fR=with\-hex option. .TP \fB\-w\fR, \fB\-\-word\-regexp\fR-The PATTERN or \fB\-e\fR \fIPATTERN\fR is searched for as a word (as if-surrounded by \\< and \\>). If PATTERN or \fB\-e\fR \fIPATTERN\fR is also-specified, then this option does not apply to \fB\-f\fR \fIFILE\fR patterns.+The PATTERN is searched for as a word (as if surrounded by \\< and+\\>). If a PATTERN is specified (or \fB\-e\fR \fIPATTERN\fR or \fB\-N\fR \fIPATTERN\fR), then+this option does not apply to \fB\-f\fR \fIFILE\fR patterns. .TP \fB\-X\fR, \fB\-\-hex\fR Output matches in hexadecimal. This option is equivalent to the \fB\-\-binary\-files\fR=hex option. .TP \fB\-x\fR, \fB\-\-line\-regexp\fR-Only input lines selected against the entire PATTERN or \fB\-e\fR \fIPATTERN\fR-is considered to be matching lines (as if surrounded by ^ and $).-If PATTERN or \fB\-e\fR \fIPATTERN\fR is also specified, then this option does+Only input lines selected against the entire PATTERN is considered+to be matching lines (as if surrounded by ^ and $). If a PATTERN+is specified (or \fB\-e\fR \fIPATTERN\fR or \fB\-N\fR \fIPATTERN\fR), then this option does not apply to \fB\-f\fR \fIFILE\fR patterns. .TP \fB\-\-xml\fR Output file matches in XML. When option \fB\-H\fR, \fB\-n\fR, \fB\-k\fR, or \fB\-b\fR is used-additional XML attributes are output. See also option \fB\-\-format\fR.+additional values are output. See also the \fB\-\-format\fR and \fB\-u\fR options. .TP \fB\-Y\fR, \fB\-\-empty\fR-Permits empty matches, such as `^\\h*$' to match blank lines. Empty-matches are disabled by default. Note that empty\-matching patterns,-such as x? and x*, match all input with \fB\-Y\fR, not only lines with `x'.+Permits empty matches. By default, empty matches are disabled,+unless a pattern starts with `^' and ends with `$'. Note that \fB\-Y\fR+when specified with an empty\-matching pattern such as x? and x*,+match all input, not only lines with a `x'. .TP \fB\-y\fR, \fB\-\-any\-line\fR Any matching or non\-matching line is output. Non\-matching lines@@ -510,17 +519,15 @@ Prints a zero\-byte after the file name. .TP \fB\-z\fR, \fB\-\-decompress\fR Decompress files to search, when compressed. Archives (.cpio,-\&.pax, .tar, .zip) and compressed archives (e.g. .taz, .tgz, .tpz,-\&.tbz, .tbz2, .tb2, .tz2, .tlz, and .txz) are searched and matching-pathnames of files in archives are output in braces. If \fB\-g\fR, \fB\-O\fR,-\fB\-M\fR, or \fB\-t\fR is specified, searches files within archives whose name-matches globs, matches file name extensions, matches file-signature magic bytes, or matches file types, respectively.-Option \fB\-\-label\fR=.ext associates suffix `ext' with standard input.-Supported compression formats: gzip (default, optional suffix .gz),-compress (requires suffix .Z), zip (requires suffix .zip or .ZIP),-bzip2 (requires suffix .bz, .bz2, or .bzip2),-lzma (requires suffix .lzma), and xz (requires suffix .xz).+\&.jar, .pax, .tar, .zip) and compressed archives (e.g. .taz, .tgz,+\&.tpz, .tbz, .tbz2, .tb2, .tz2, .tlz, and .txz) are searched and+matching pathnames of files in archives are output in braces. If+\fB\-g\fR, \fB\-O\fR, \fB\-M\fR, or \fB\-t\fR is specified, searches files within archives+whose name matches globs, matches file name extensions, matches+file signature magic bytes, or matches file types, respectively.+Supported compression formats: gzip (.gz), compress (.Z), zip,+bzip2 (requires suffix .bz, .bz2, .bzip2, .tbz, .tbz2, .tb2, .tz2),+lzma and xz (requires suffix .lzma, .tlz, .xz, .txz). .PP If no \fIFILE\fR arguments are specified, or if a `-' is specified, the standard input is used, unless recursive searches are specified which examine@@ -542,7 +549,7 @@ https://github.com/Genivia/ugrep Note that `.' matches any non-newline character. Pattern `\\n' matches a newline character. Multiple lines may be matched with patterns that match newlines, unless one or more of the context options \fB-A\fR, \fB-B\fR,- \fB-C\fR or \fB-y\fR is used, or option \fB-v\fR.+\fB-C\fR, or \fB-y\fR is used, or option \fB-v\fR is used. .SH "EXIT STATUS" The \fBugrep\fR utility exits with one of the following values: .IP 0@@ -563,8 +570,11 @@ a glob contains a path separator `/', the pathname is matched. Otherwise the basename of a file or directory is matched. For example, \fB*.h\fR matches \fIfoo.h\fR and \fIbar/foo.h\fR. \fBbar/*.h\fR matches \fIbar/foo.h\fR but not \fIfoo.h\fR and not \fIbar/bar/foo.h\fR. Use a leading `/' to force \fB/*.h\fR-to match \fIfoo.h\fR but not \fIbar/foo.h\fR. A glob starting with a `!' is-negated, i.e. does not match.+to match \fIfoo.h\fR but not \fIbar/foo.h\fR.+.PP+When a glob starts with a `!' as specified with \fB-g\fR!\fIGLOB\fR, or+specified in a \fIFILE\fR with \fB--include-from\fR=\fIFILE\fR or+\fB--exclude-from\fR=\fIFILE\fR, it is negated. .PP \fBGlob Syntax and Conventions\fR .IP \fB*\fR@@ -636,9 +646,9 @@ list of ANSI SGR parameters that defaults to \fBms=\fR, and \fBmc=\fR capabilities of \fBGREP_COLORS\fR have priority over \fBGREP_COLOR\fR. Option \fB--colors\fR has priority over \fBGREP_COLORS\fR. .SH GREP_COLORS-Colors are specified as a substring of semicolon-separated ANSI SGR codes to-highlight specific parts of the output when option \fB--color=auto\fR or-\fB--color=always\fR is used:+Colors are specified as string of colon-separated ANSI SGR parameters of the+form `what=substring', where `substring' is a semicolon-separated list of SGR+codes to color-highlight `what': .IP \fBsl=\fR SGR substring for selected lines. .IP \fBcx=\fR@@ -750,22 +760,21 @@ if not the first match: a verical bar, same as \fB%[|]>\fR. the percentage sign. .IP \fB%1\fR the first regex group capture of the match, and so on up to group \fB%9\fR,-same as \fB%[1]#\fR, requires option \fB-P\fR Perl matching.+same as \fB%[1]#\fR; requires option \fB-P\fR Perl matching. .IP \fB%[\fINUM\fR\fB]#\fR-the regex group capture \fINUM\fR, requires option \fB-P\fR.+the regex group capture \fINUM\fR; requires option \fB-P\fR Perl matching. .PP The \fB[\fR\fIARG\fR\fB]\fR part of a field is optional and may be omitted. When present, the argument must be placed in \fB[]\fR brackets, for example \fB%[,]F\fR to output a comma, the pathname, and a separator. .PP-Fields \fB%[\fR\fISEP\fR\fB]$\fR and \fB%u\fR are switches and do not write+Fields \fB%[\fR\fISEP\fR\fB]$\fR and \fB%u\fR are switches and do not send anything to the output. .PP The separator used by \fB%P\fR, \fB%H\fR, \fB%N\fR, \fB%K\fR, \fB%B\fR, and-\fB%S\fR may be changed by preceeding the field with a-\fB%[\fR\fISEP\fR\fB]$\fR. When \fB[\fR\fISEP\fR\fB]\fR is not provided,-reverses the separator to the default separator or the separator specified by-\fB--separator\fR.+\fB%S\fR may be changed by preceeding the field by \fB%[\fR\fISEP\fR\fB]$\fR.+When \fB[\fR\fISEP\fR\fB]\fR is not provided, this reverses the separator to+the default separator or the separator specified with \fB--separator\fR. .PP Formatted output is written for each matching pattern, which means that a line may be output multiple times when patterns match more than once on the same@@ -862,7 +871,7 @@ Find lines with `FIXME' in the C/C++ files stored in a tarball: .IP $ ugrep -z -tc++ -n 'FIXME' project.tgz .PP-Recursively search for the word `copyright' in cpio/pax/tar/zip archives,+Recursively search for the word `copyright' in cpio/jar/pax/tar/zip archives, compressed and regular files, and in PDFs using a PDF filter: .IP $ ugrep -r -z -w --filter='pdf:pdftotext % -' 'copyright'
@@ -50,19 +50,19 @@ Optional libraries to support options -P and -z: Build ugrep as follows: - ./configure- make+ ./configure && make++Or build ugrep with colors enabled by default:++ ./configure --enable-color && make Github does not preserve time stamps so ./configure may fail, in that case do: ./autoreconf -fi- ./configure- make+ ./configure && make -Install as follows:+After this, you may want to install ugrep (optional): - ./configure- make sudo make install Prebuilt executables are located in ugrep/bin.@@ -70,7 +70,7 @@ Prebuilt executables are located in ugrep/bin. */ // ugrep version-#define UGREP_VERSION "1.7.2"+#define UGREP_VERSION "1.7.3" #include <reflex/input.h> #include <reflex/matcher.h>@@ -329,7 +329,7 @@ int pclose(FILE *stream) // undefined size_t value #define UNDEFINED static_cast<size_t>(~0UL) -// the -M MAGIC pattern (compiled, read-only)+// the -M MAGIC pattern DFAs constructed before threads start, read-only afterwards reflex::Pattern magic_pattern; // number of concurrent threads for workers@@ -439,6 +439,8 @@ size_t flag_max_depth = 0; size_t flag_max_files = 0; size_t flag_min_line = 0; size_t flag_max_line = 0;+size_t flag_not_magic = 0;+size_t flag_min_magic = 1; size_t flag_jobs = 0; size_t flag_tabs = 8; size_t flag_min_mmap = DEFAULT_MIN_MMAP_SIZE;@@ -747,27 +749,32 @@ struct Stats { for (auto& i : ignore) fprintf(output, " %s exclusions were applied to %s\n", i.c_str(), i.substr(0, i.find_last_of(PATHSEPCHR)).c_str()); for (auto& i : flag_file_magic)- fprintf(output, "--file-magic='%s'\n", i.c_str());+ {+ if (!i.empty() && (i.front() == '!' || i.front() == '^'))+ fprintf(output, "--file-magic='!%s' (negation)\n", i.c_str() + 1);+ else+ fprintf(output, "--file-magic='%s'\n", i.c_str());+ } for (auto& i : flag_include) fprintf(output, "--include='%s'\n", i.c_str()); for (auto& i : flag_not_include)- fprintf(output, "--include='!%s' (negation override)\n", i.c_str());+ fprintf(output, "--include='!%s' (negation)\n", i.c_str()); for (auto& i : flag_include_fs) fprintf(output, "--include-fs='%s'\n", i.c_str()); for (auto& i : flag_include_dir) fprintf(output, "--include-dir='%s'\n", i.c_str()); for (auto& i : flag_not_include_dir)- fprintf(output, "--include-dir='!%s' (negation override)\n", i.c_str());+ fprintf(output, "--include-dir='!%s' (negation)\n", i.c_str()); for (auto& i : flag_exclude) fprintf(output, "--exclude='%s'\n", i.c_str()); for (auto& i : flag_not_exclude)- fprintf(output, "--exclude='!%s' (negation override)\n", i.c_str());+ fprintf(output, "--exclude='!%s' (negation)\n", i.c_str()); for (auto& i : flag_exclude_fs) fprintf(output, "--exclude-fs='%s'\n", i.c_str()); for (auto& i : flag_exclude_dir) fprintf(output, "--exclude-dir='%s'\n", i.c_str()); for (auto& i : flag_not_exclude_dir)- fprintf(output, "--exclude-dir='!%s' (negation override)\n", i.c_str());+ fprintf(output, "--exclude-dir='!%s' (negation)\n", i.c_str()); } } @@ -2941,15 +2948,13 @@ struct Grep { } } - // -M: check magic bytes, requires sufficiently large len of buf[]- if ((flag_include.empty() || !is_selected) && !flag_file_magic.empty())+ // -M: check magic bytes, requires sufficiently large len of buf[] to match patterns, which is fine when Z_BUF_LEN is large e.g. 64K+ if (buf != NULL && !flag_file_magic.empty() && (flag_include.empty() || !is_selected)) {- if (buf != NULL)- {- reflex::Matcher magic(magic_pattern);- magic.buffer(const_cast<char*>(reinterpret_cast<const char*>(buf)), len + 1);- is_selected = magic.scan() != 0;- }+ reflex::Matcher magic(magic_pattern);+ magic.buffer(const_cast<char*>(reinterpret_cast<const char*>(buf)), len + 1);+ size_t match = magic.scan();+ is_selected = match == flag_not_magic || match >= flag_min_magic; } } @@ -4155,6 +4160,10 @@ int main(int argc, char **argv) if (flag_regexp.empty() && flag_file.empty()) help(""); + // -x: enable -Y+ if (flag_line_regexp)+ flag_empty = true;+ // -F: make newline-separated lines in regex literal with \Q and \E const char *Q = flag_fixed_strings ? "\\Q" : ""; const char *E = flag_fixed_strings ? "\\E|" : "|";@@ -4174,14 +4183,14 @@ int main(int argc, char **argv) { regex.append(".*\n?|"); - // indicate we're matching everything+ // we're matching everything flag_match = true; } else { regex.append(".*|"); - // indicate we're matching everything+ // we're matching everything flag_match = true; } @@ -4209,8 +4218,9 @@ int main(int argc, char **argv) if (from < pattern.size()) regex.append(Q).append(pattern.substr(from)).append(E); - if (pattern == "^$")- flag_empty = true; // we're matching empty lines, so enable -Y+ // if pattern starts with ^ and ends with $, enable -Y+ if (pattern.size() >= 2 && pattern.front() == '^' && pattern.back() == '$')+ flag_empty = true; } } @@ -4240,6 +4250,9 @@ int main(int argc, char **argv) if (from < pattern.size()) neg_regex.append(Q).append(pattern.substr(from)).append(E);++ if (pattern.size() >= 2 && pattern.front() == '^' && pattern.back() == '$')+ flag_empty = true; // we're possibly matching empty lines, so enable -Y } } @@ -4249,11 +4262,14 @@ int main(int argc, char **argv) // remove the ending '|' from the |-concatenated regexes in the regex string neg_regex.pop_back(); - // -x or -w- if (flag_line_regexp)- neg_regex.insert(0, "^(").append(")$"); // make the regex line-anchored- else if (flag_word_regexp)- neg_regex.insert(0, "\\<(").append(")\\>"); // make the regex word-anchored+ if (regex != "^$")+ {+ // -x or -w+ if (flag_line_regexp)+ neg_regex.insert(0, "^(").append(")$"); // make the regex line-anchored+ else if (flag_word_regexp)+ neg_regex.insert(0, "\\<(").append(")\\>"); // make the regex word-anchored+ } // construct negative (?^PATTERN) neg_regex.insert(0, "(?^").push_back(')');@@ -4677,22 +4693,49 @@ int main(int argc, char **argv) { size_t to = types.find(',', from); size_t size = (to == std::string::npos ? types.size() : to) - from;- std::string type(types.substr(from, size)); - size_t i;+ if (size > 0)+ {+ bool negate = size > 1 && (types[from] == '!' || types[from] == '^'); - // scan the type_table[] for a matching type- for (i = 0; type_table[i].type != NULL; ++i)- if (type == type_table[i].type)- break;+ if (negate)+ {+ ++from;+ --size;+ } - if (type_table[i].type == NULL)- help("invalid argument --file-type=TYPE, to list the valid values use -tlist");+ std::string type(types.substr(from, size)); - flag_file_extensions.emplace_back(type_table[i].extensions);+ size_t i; - if (type_table[i].magic != NULL)- flag_file_magic.emplace_back(type_table[i].magic);+ // scan the type_table[] for a matching type+ for (i = 0; type_table[i].type != NULL; ++i)+ if (type == type_table[i].type)+ break;++ if (type_table[i].type == NULL)+ help("invalid argument --file-types=TYPES, -tlist displays valid values");++ std::string extensions(type_table[i].extensions);++ if (negate)+ {+ extensions.insert(0, "!");+ size_t j = 0;+ while ((j = extensions.find(',', j)) != std::string::npos)+ extensions.insert(++j, "!");+ }++ flag_file_extensions.emplace_back(extensions);++ if (type_table[i].magic != NULL)+ {+ flag_file_magic.emplace_back(type_table[i].magic);++ if (negate)+ flag_file_magic.back().insert(0, "!");+ }+ } if (to == std::string::npos) break;@@ -4701,17 +4744,18 @@ int main(int argc, char **argv) } } - // -g, --glob: add globs to --include and --exclude+ // -g, --glob: add globs to --include/--exclude for (auto& i : flag_glob) {- bool negate = !i.empty() && i.front() == '!';+ bool negate = i.size() > 1 && (i.front() == '!' || i.front() == '^');+ if (negate) flag_exclude.emplace_back(i.substr(1)); else flag_include.emplace_back(i); } - // -O: add extensions as globs to the --include list+ // -O: add filename extensions as globs to --include/--exclude for (auto& extensions : flag_file_extensions) { size_t from = 0;@@ -4722,27 +4766,38 @@ int main(int argc, char **argv) size_t to = extensions.find(',', from); size_t size = (to == std::string::npos ? extensions.size() : to) - from; - flag_include.emplace_back(glob.assign("*.").append(extensions.substr(from, size)));+ if (size > 0)+ {+ bool negate = size > 1 && (extensions[from] == '!' || extensions[from] == '^');++ if (negate)+ {+ ++from;+ --size;+ }++ (negate ? flag_exclude : flag_include).emplace_back(glob.assign("*.").append(extensions.substr(from, size))); #ifdef HAVE_LIBZ- // -z: add globs to search compressed files and tarballs- if (flag_decompress)- {- const char *zextensions[] = {- ".gz", ".Z", ".zip", ".ZIP",+ // -z: add globs to search compressed files and tarballs+ if (!negate && flag_decompress)+ {+ const char *zextensions[] = {+ ".gz", ".Z", ".zip", ".ZIP", #ifdef HAVE_LIBBZ2- ".bz", ".bz2", ".bzip2",+ ".bz", ".bz2", ".bzip2", #endif #ifdef HAVE_LIBLZMA- ".lzma", ".xz",+ ".lzma", ".xz", #endif- NULL- };+ NULL+ }; - for (size_t i = 0; zextensions[i] != NULL; ++i)- flag_include.emplace_back(glob.assign("*.").append(extensions.substr(from, size)).append(zextensions[i]));- }+ for (size_t i = 0; zextensions[i] != NULL; ++i)+ flag_include.emplace_back(glob.assign("*.").append(extensions.substr(from, size)).append(zextensions[i]));+ } #endif+ } if (to == std::string::npos) break;@@ -4807,15 +4862,35 @@ int main(int argc, char **argv) #endif #endif - // -M: file signature "magic bytes" MAGIC regex+ // -M: file signature "magic bytes" regex string std::string signature; - // -M: combine to create a signature regex from MAGIC- for (auto& magic : flag_file_magic)+ // -M !MAGIC: combine to create a signature regex string+ for (auto& i : flag_file_magic) {- if (!signature.empty())- signature.push_back('|');- signature.append(magic);+ if (i.size() > 1 && (i.front() == '!' || i.front() == '^'))+ {+ if (!signature.empty())+ signature.push_back('|');+ signature.append(i.substr(1));++ // tally negative MAGIC patterns+ ++flag_min_magic;+ }+ }++ // -M MAGIC: append to signature regex string+ for (auto& i : flag_file_magic)+ {+ if (i.size() <= 1 || (i.front() != '!' && i.front() != '^'))+ {+ if (!signature.empty())+ signature.push_back('|');+ signature.append(i);++ // we have positive MAGIC patterns, so scan() is a match when flag_min_magic or greater+ flag_not_magic = flag_min_magic;+ } } // --exclude-from: add globs to the --exclude and --exclude-dir lists@@ -4851,12 +4926,16 @@ int main(int argc, char **argv) { size_t to = i.find(',', from); size_t size = (to == std::string::npos ? i.size() : to) - from;- std::string mount(i.substr(from, size)); - if (statvfs(mount.c_str(), &buf) == 0)- exclude_fs.insert(static_cast<uint64_t>(buf.f_fsid));- else- warning("--exclude-fs: cannot stat", mount.c_str());+ if (size > 0)+ {+ std::string mount(i.substr(from, size));++ if (statvfs(mount.c_str(), &buf) == 0)+ exclude_fs.insert(static_cast<uint64_t>(buf.f_fsid));+ else+ warning("--exclude-fs: cannot stat", mount.c_str());+ } if (to == std::string::npos) break;@@ -4901,12 +4980,16 @@ int main(int argc, char **argv) { size_t to = i.find(',', from); size_t size = (to == std::string::npos ? i.size() : to) - from;- std::string mount(i.substr(from, size)); - if (statvfs(mount.c_str(), &buf) == 0)- include_fs.insert(static_cast<uint64_t>(buf.f_fsid));- else- warning("--include-fs: cannot stat", mount.c_str());+ if (size > 0)+ {+ std::string mount(i.substr(from, size));++ if (statvfs(mount.c_str(), &buf) == 0)+ include_fs.insert(static_cast<uint64_t>(buf.f_fsid));+ else+ warning("--include-fs: cannot stat", mount.c_str());+ } if (to == std::string::npos) break;@@ -4982,7 +5065,9 @@ int main(int argc, char **argv) try {- magic_pattern.assign(signature, "r");+ // construct magic_pattern DFA for -M !MAGIC and -M MAGIC+ if (!signature.empty())+ magic_pattern.assign(signature, "r"); magic.pattern(magic_pattern); } @@ -5329,7 +5414,8 @@ void find(size_t level, reflex::Matcher& magic, Grep& grep, const char *pathname std::istream stream(&streambuf); // file has the magic bytes we're looking for: search the file- if (magic.input(&stream).scan() != 0)+ size_t match = magic.input(&stream).scan();+ if (match == flag_not_magic || match >= flag_min_magic) { stats.score_file(); @@ -5342,16 +5428,19 @@ void find(size_t level, reflex::Matcher& magic, Grep& grep, const char *pathname } else #endif- if (magic.input(reflex::Input(file, flag_encoding_type)).scan() != 0) {- // if file has the magic bytes we're looking for: search the file- stats.score_file();+ size_t match = magic.input(reflex::Input(file, flag_encoding_type)).scan();+ if (match == flag_not_magic || match >= flag_min_magic)+ {+ // if file has the magic bytes we're looking for: search the file+ stats.score_file(); - fclose(file);+ fclose(file); - grep.search(pathname);+ grep.search(pathname); - return;+ return;+ } } fclose(file);@@ -5492,7 +5581,8 @@ void find(size_t level, reflex::Matcher& magic, Grep& grep, const char *pathname std::istream stream(&streambuf); // file has the magic bytes we're looking for: search the file- if (magic.input(&stream).scan() != 0)+ size_t match = magic.input(&stream).scan();+ if (match == flag_not_magic || match >= flag_min_magic) { stats.score_file(); @@ -5507,7 +5597,8 @@ void find(size_t level, reflex::Matcher& magic, Grep& grep, const char *pathname #endif { // if file has the magic bytes we're looking for: search the file- if (magic.input(reflex::Input(file, flag_encoding_type)).scan() != 0)+ size_t match = magic.input(reflex::Input(file, flag_encoding_type)).scan();+ if (match == flag_not_magic || match >= flag_min_magic) { stats.score_file(); @@ -6241,7 +6332,7 @@ void Grep::search(const char *pathname) if (out.eof) goto exit_search; - const char *eol = matcher->eol(true); // warning: call eol() before bol()+ const char *eol = matcher->eol(true); // warning: call eol() before bol() and end() const char *bol = matcher->bol(); binary = flag_hex || (!flag_text && is_binary(bol, eol - bol));@@ -6429,7 +6520,7 @@ void Grep::search(const char *pathname) } else {- const char *eol = matcher->eol(true); // warning: call eol() before bol()+ const char *eol = matcher->eol(true); // warning: call eol() before end() const char *end = matcher->end(); bool rest_binary = flag_hex || (!flag_text && is_binary(end, eol - end));@@ -7199,6 +7290,7 @@ void extend(FILE *file, std::vector<std::string>& files, std::vector<std::string { // gitignore-style ! negate pattern to override include/exclude bool negate = line.front() == '!' && !line.empty();+ if (negate) line.erase(0, 1); @@ -7565,8 +7657,8 @@ void help(const char *message, const char *arg) zero byte or invalid UTF encoding. See also the -a, -I, -U, -W,\n\ and -X options.\n\ --break\n\- Groups matches per file and adds a line break between results from\n\- different files.\n\+ Group matches per file. Adds a header and line break between\n\+ results from different files.\n\ -C[NUM], --context[=NUM]\n\ Print NUM lines of leading and trailing context surrounding each\n\ match. The default is 2 and is equivalent to -A 2 -B 2. Places\n\@@ -7587,10 +7679,10 @@ void help(const char *message, const char *arg) list of ANSI SGR parameters. COLORS selectively overrides the\n\ colors specified by the GREP_COLORS environment variable.\n\ --cpp\n\- Output file matches in C++. See also option --format.\n\+ Output file matches in C++. See also the --format and -u options.\n\ --csv\n\ Output file matches in CSV. When option -H, -n, -k, or -b is used\n\- additional CSV values are output. See also option --format.\n\+ additional values are output. See also the --format and -u options.\n\ -D ACTION, --devices=ACTION\n\ If an input file is a device, FIFO or socket, use ACTION to process\n\ it. By default, ACTION is `skip', which means that devices are\n\@@ -7619,29 +7711,31 @@ void help(const char *message, const char *arg) --exclude=GLOB\n\ Skip files whose name matches GLOB using wildcard matching, same as\n\ -g !GLOB. GLOB can use **, *, ?, and [...] as wildcards, and \\ to\n\- quote a wildcard or backslash character literally. If GLOB\n\- contains /, full pathnames are matched. Otherwise basenames are\n\- matched. Note that --exclude patterns take priority over --include\n\- patterns. This option may be repeated.\n\+ quote a wildcard or backslash character literally. When GLOB\n\+ contains a `/', full pathnames are matched. Otherwise basenames\n\+ are matched. Note that --exclude patterns take priority over\n\+ --include patterns. This option may be repeated.\n\ --exclude-dir=GLOB\n\ Exclude directories whose name matches GLOB from recursive\n\- searches. If GLOB contains /, full pathnames are matched.\n\- Otherwise basenames are matched. Note that --exclude-dir patterns\n\- take priority over --include-dir patterns. This option may be\n\- repeated.\n\+ searches. GLOB can use **, *, ?, and [...] as wildcards, and \\ to\n\+ quote a wildcard or backslash character literally. When GLOB\n\+ contains a `/', full pathnames are matched. Otherwise basenames\n\+ are matched. Note that --exclude-dir patterns take priority over\n\+ --include-dir patterns. This option may be repeated.\n\ --exclude-from=FILE\n\ Read the globs from FILE and skip files and directories whose name\n\ matches one or more globs (as if specified by --exclude and\n\ --exclude-dir). Lines starting with a `#' and empty lines in FILE\n\ are ignored. When FILE is a `-', standard input is read. This\n\- option may be repeated.\n";-#ifdef HAVE_STATVFS- std::cout << "\+ option may be repeated.\n\ --exclude-fs=MOUNTS\n\- Exclude file systems with mounting points MOUNTS from recursive\n\- searches, MOUNTS is a comma-separated list of pathnames. Note\n\- that --exclude-fs mounts take priority over --include-fs mounts.\n\- This option may be repeated.\n";+ Exclude file systems specified by MOUNTS from recursive searches,\n\+ MOUNTS is a comma-separated list of mount points or pathnames of\n\+ directories on file systems. Note that --exclude-fs mounts take\n\+ priority over --include-fs mounts. This option may be repeated.\n";+#ifndef HAVE_STATVFS+ std::cout << "\+ This option is not available in this build configuration of ugrep.\n"; #endif std::cout << "\ -F, --fixed-strings\n\@@ -7669,11 +7763,11 @@ void help(const char *message, const char *arg) `command' is a filter utility. The filter utility should read from\n\ standard input and write to standard output. Files matching one of\n\ `exts` are filtered only. One or more `option' separated by\n\- spacing may be specified, which are passed verbatim to the command\n\- without expansion. A `%' as `option' expands into the pathname,\n\- which is the conventional dash `-' when searching standard input.\n\- Option --label=.ext may be used to force filtering standard input.\n\- For example, --filter='pdf:pdftotext % -' searches PDF files.\n";+ spacing may be specified, which are passed verbatim to the command.\n\+ A `%' as `option' expands into the pathname to search. For example,\n\+ --filter='pdf:pdftotext % -' searches PDF files. The `%' expands\n\+ into a `-' when searching standard input. Option --label=.ext may\n\+ be used to specify extension `ext' when searching standard input.\n"; #endif std::cout << "\ --format=FORMAT\n\@@ -7686,8 +7780,8 @@ void help(const char *message, const char *arg) behave as traditional grep.\n\ -g GLOB, --glob=GLOB\n\ Search only files whose name matches GLOB, same as --include=GLOB.\n\- If GLOB is preceded by a `!', skip files whose name matches GLOB,\n\- same as --exclude=GLOB.\n\+ When GLOB is preceded by a `!' or a `^', skip files whose name\n\+ matches GLOB, same as --exclude=GLOB.\n\ --group-separator[=SEP]\n\ Use SEP as a group separator for context options -A, -B, and -C.\n\ The default is a double hyphen (`--').\n\@@ -7706,38 +7800,43 @@ void help(const char *message, const char *arg) Perform case insensitive matching. By default, ugrep is case\n\ sensitive. This option applies to ASCII letters only.\n\ --ignore-files[=FILE]\n\- Ignore files and directories specified in a FILE when encountered\n\- in recursive searches. The default is `" DEFAULT_IGNORE_FILE "'. Files and\n\- directories matching the globs in FILE are ignored in the directory\n\- tree rooted at each FILE's location by temporarily overriding\n\- --exclude and --exclude-dir globs. This option may be repeated.\n\+ Ignore files and directories matching the globs in each FILE when\n\+ encountered in recursive searches. The default FILE is\n\+ `" DEFAULT_IGNORE_FILE "'. Matching files and directories located in the\n\+ directory tree rooted at a FILE's location are ignored by\n\+ temporarily overriding the --exclude and --exclude-dir globs.\n\+ Files and directories specified as FILE arguments are not ignored.\n\+ This option may be repeated.\n\ --include=GLOB\n\ Search only files whose name matches GLOB using wildcard matching,\n\ same as -g GLOB. GLOB can use **, *, ?, and [...] as wildcards,\n\- and \\ to quote a wildcard or backslash character literally. If\n\- GLOB contains /, file pathnames are matched. Otherwise file\n\+ and \\ to quote a wildcard or backslash character literally. When\n\+ GLOB contains a `/', full pathnames are matched. Otherwise\n\ basenames are matched. Note that --exclude patterns take priority\n\ over --include patterns. This option may be repeated.\n\ --include-dir=GLOB\n\ Only directories whose name matches GLOB are included in recursive\n\- searches. If GLOB contains /, full pathnames are matched.\n\- Otherwise basenames are matched. Note that --exclude-dir patterns\n\- take priority over --include-dir patterns. This option may be\n\- repeated.\n\+ searches. GLOB can use **, *, ?, and [...] as wildcards, and \\ to\n\+ quote a wildcard or backslash character literally. When GLOB\n\+ contains a `/', full pathnames are matched. Otherwise basenames\n\+ are matched. Note that --exclude-dir patterns take priority over\n\+ --include-dir patterns. This option may be repeated.\n\ --include-from=FILE\n\ Read the globs from FILE and search only files and directories\n\ whose name matches one or more globs (as if specified by --include\n\ and --include-dir). Lines starting with a `#' and empty lines in\n\ FILE are ignored. When FILE is a `-', standard input is read.\n\- This option may be repeated.\n";-#ifdef HAVE_STATVFS- std::cout << "\+ This option may be repeated.\n\ --include-fs=MOUNTS\n\- Only file systems with mounting points MOUNTS are included in\n\- recursive searches. MOUNTS is a comma-separated list of pathnames.\n\- --include-fs=. restricts recursive searches to the file system of\n\- the working directory only. Note that --exclude-fs mounts take\n\- priority over --include-fs mounts. This option may be repeated.\n";+ Only file systems specified by MOUNTS are included in recursive\n\+ searches. MOUNTS is a comma-separated list of mount points or\n\+ pathnames of directories on file systems. --include-fs=. restricts\n\+ recursive searches to the file system of the working directory\n\+ only. Note that --exclude-fs mounts take priority over\n\+ --include-fs mounts. This option may be repeated.\n";+#ifndef HAVE_STATVFS+ std::cout << "\+ This option is not available in this build configuration of ugrep.\n"; #endif std::cout << "\ -J NUM, --jobs=NUM\n\@@ -7750,13 +7849,13 @@ void help(const char *message, const char *arg) letter. Case insensitive matching applies to ASCII letters only.\n\ --json\n\ Output file matches in JSON. When option -H, -n, -k, or -b is used\n\- additional JSON properties are output. See also option --format.\n\+ additional values are output. See also the --format and -u options.\n\ -K NUM1[,NUM2], --range=NUM1[,NUM2]\n\ Start searching at line NUM1 and end at line NUM2 when specified.\n\ -k, --column-number\n\ The column number of a matched pattern is displayed in front of the\n\ respective matched line, starting at column 1. Tabs are expanded\n\- when columns are counted, see option --tabs.\n\+ when columns are counted, see also option --tabs.\n\ -L, --files-without-match\n\ Only the names of files not containing selected lines are written\n\ to standard output. Pathnames are listed once per file searched.\n\@@ -7779,10 +7878,11 @@ void help(const char *message, const char *arg) -M MAGIC, --file-magic=MAGIC\n\ Only files matching the signature pattern MAGIC are searched. The\n\ signature \"magic bytes\" at the start of a file are compared to\n\- the `MAGIC' regex pattern. When matching, the file will be\n\- searched. This option may be repeated and may be combined with\n\- options -O and -t to expand the search. This option is relatively\n\- slow as every file on the search path is read to compare MAGIC.\n\+ the MAGIC regex pattern. When matching, the file will be searched.\n\+ When MAGIC is preceded by a `!' or a `^', skip files with matching\n\+ MAGIC signatures. This option may be repeated and may be combined\n\+ with options -O and -t to expand the search. Every file on the\n\+ search path is read, making searches potentially more expensive.\n\ -m NUM, --max-count=NUM\n\ Stop reading the input after NUM matches for each file processed.\n\ --match\n\@@ -7797,9 +7897,9 @@ void help(const char *message, const char *arg) NUM. Specify -J1 to produce replicable results by ensuring that\n\ files are searched in the same order as specified.\n\ -N PATTERN, --neg-regexp=PATTERN\n\- Specify a negative PATTERN used during the search of the input: an\n\- input line is selected only if it matches any of the specified\n\- patterns when PATTERN does not match. Same as -e (?^PATTERN)\n\+ Specify a negative PATTERN used during the search of the input:\n\+ an input line is selected only if it matches any of the specified\n\+ patterns when PATTERN does not match. Same as -e (?^PATTERN).\n\ Negative PATTERN matches are removed before any other specified\n\ patterns are matched. Note that longer patterns take precedence\n\ over shorter patterns. This option may be repeated.\n\@@ -7825,12 +7925,13 @@ void help(const char *message, const char *arg) std::cout << "\ -O EXTENSIONS, --file-extensions=EXTENSIONS\n\ Search only files whose filename extensions match the specified\n\- comma-separated list of EXTENSIONS. This option is the same as\n\- specifying --include='*.ext' for each extension suffix `ext' in the\n\- EXTENSIONS list. This option may be repeated and may be combined\n\+ comma-separated list of EXTENSIONS, same as --include='*.ext' for\n\+ each `ext' in EXTENSIONS. When `ext' is preceded by a `!' or a\n\+ `^', skip files whose filename extensions matches `ext', same as\n\+ --exclude='*.ext'. This option may be repeated and may be combined\n\ with options -M and -t to expand the recursive search.\n\ -o, --only-matching\n\- Prints only the matching part of lines. When multiple lines match,\n\+ Print only the matching part of lines. When multiple lines match,\n\ the line numbers with option -n are displayed using `|' as the\n\ field separator for each additional line matched by the pattern.\n\ This option cannot be combined with options -A, -B, -C, -v, and -y.\n\@@ -7842,7 +7943,7 @@ void help(const char *message, const char *arg) Interpret PATTERN as a Perl regular expression.\n"; #ifndef HAVE_BOOST_REGEX std::cout << "\- This feature is not available in this build configuration of ugrep.\n";+ This option is not available in this build configuration of ugrep.\n"; #endif std::cout << "\ -p, --no-dereference\n\@@ -7863,11 +7964,11 @@ void help(const char *message, const char *arg) match has been found, making searches potentially less expensive.\n\ -R, --dereference-recursive\n\ Recursively read all files under each directory. Follow all\n\- symbolic links, unlike -r. If -J1 is specified, files are searched\n\- in the same order as specified.\n\+ symbolic links, unlike -r. When -J1 is specified, files are\n\+ searched in the same order as specified.\n\ -r, --recursive\n\ Recursively read all files under each directory, following symbolic\n\- links only if they are on the command line. If -J1 is specified,\n\+ links only if they are on the command line. When -J1 is specified,\n\ files are searched in the same order as specified.\n\ -S, --dereference\n\ If -r is specified, all symbolic links are followed, like -R. The\n\@@ -7881,18 +7982,19 @@ void help(const char *message, const char *arg) (`:').\n\ --stats\n\ Display statistics on the number of files and directories searched.\n\- Display MAGIC and GLOB selections applied to recursive searches.\n\+ Display the inclusion and exclusion constraints applied.\n\ -T, --initial-tab\n\ Add a tab space to separate the file name, line number, column\n\ number, and byte offset with the matched line.\n\ -t TYPES, --file-type=TYPES\n\ Search only files associated with TYPES, a comma-separated list of\n\- file types. Each file type corresponds to a set of file name\n\+ file types. Each file type corresponds to a set of filename\n\ extensions passed to option -O. For capitalized file types, the\n\- search is expanded to include files found on the search path with\n\- matching file signature magic bytes passed to option -M. This\n\- option may be repeated. The possible values of TYPES can be\n\- (use option -tlist to display a more detailed list):";+ search is expanded to include files with matching file signature\n\+ magic bytes, as if passed to option -M. When a type is preceeded\n\+ by a `!' or a `^', excludes files of the specified type. This\n\+ option may be repeated. The possible file types can be (where\n\+ -tlist displays a detailed list):"; for (int i = 0; type_table[i].type != NULL; ++i) std::cout << (i == 0 ? "" : ",") << (i % 7 ? " " : "\n ") << "`" << type_table[i].type << "'"; std::cout << ".\n\@@ -7917,24 +8019,25 @@ void help(const char *message, const char *arg) Output binary matches in hexadecimal, leaving text matches alone.\n\ This option is equivalent to the --binary-files=with-hex option.\n\ -w, --word-regexp\n\- The PATTERN or -e PATTERN is searched for as a word (as if\n\- surrounded by \\< and \\>). If PATTERN or -e PATTERN is also\n\- specified, then this option does not apply to -f FILE patterns.\n\+ The PATTERN is searched for as a word (as if surrounded by \\< and\n\+ \\>). If a PATTERN is specified (or -e PATTERN or -N PATTERN), then\n\+ this option does not apply to -f FILE patterns.\n\ -X, --hex\n\ Output matches in hexadecimal. This option is equivalent to the\n\ --binary-files=hex option.\n\ -x, --line-regexp\n\- Only input lines selected against the entire PATTERN or -e PATTERN\n\- is considered to be matching lines (as if surrounded by ^ and $).\n\- If PATTERN or -e PATTERN is also specified, then this option does\n\+ Only input lines selected against the entire PATTERN is considered\n\+ to be matching lines (as if surrounded by ^ and $). If a PATTERN\n\+ is specified (or -e PATTERN or -N PATTERN), then this option does\n\ not apply to -f FILE patterns.\n\ --xml\n\ Output file matches in XML. When option -H, -n, -k, or -b is used\n\- additional XML attributes are output. See also option --format.\n\+ additional values are output. See also the --format and -u options.\n\ -Y, --empty\n\- Permits empty matches, such as `^\\h*$' to match blank lines. Empty\n\- matches are disabled by default. Note that empty-matching patterns,\n\- such as x? and x*, match all input with -Y, not only lines with `x'.\n\+ Permits empty matches. By default, empty matches are disabled,\n\+ unless a pattern starts with `^' and ends with `$'. Note that -Y\n\+ when specified with an empty-matching pattern such as x? and x*,\n\+ match all input, not only lines with a `x'.\n\ -y, --any-line\n\ Any matching or non-matching line is output. Non-matching lines\n\ are output with the `-' separator as context of the matching lines.\n\@@ -7943,27 +8046,25 @@ void help(const char *message, const char *arg) Prints a zero-byte after the file name.\n\ -z, --decompress\n\ Decompress files to search, when compressed. Archives (.cpio,\n\- .pax, .tar, .zip) and compressed archives (e.g. .taz, .tgz, .tpz,\n\- .tbz, .tbz2, .tb2, .tz2, .tlz, and .txz) are searched and matching\n\- pathnames of files in archives are output in braces. If -g, -O,\n\- -M, or -t is specified, searches files within archives whose name\n\- matches globs, matches file name extensions, matches file\n\- signature magic bytes, or matches file types, respectively.\n";+ .jar, .pax, .tar, .zip) and compressed archives (e.g. .taz, .tgz,\n\+ .tpz, .tbz, .tbz2, .tb2, .tz2, .tlz, and .txz) are searched and\n\+ matching pathnames of files in archives are output in braces. If\n\+ -g, -O, -M, or -t is specified, searches files within archives\n\+ whose name matches globs, matches file name extensions, matches\n\+ file signature magic bytes, or matches file types, respectively.\n"; #ifndef HAVE_LIBZ std::cout << "\- This feature is not available in this build configuration of ugrep.\n";+ This option is not available in this build configuration of ugrep.\n"; #else std::cout << "\- Option --label=.ext associates suffix `ext' with standard input.\n\- Supported compression formats: gzip (default, optional suffix .gz),\n\- compress (requires suffix .Z), zip (requires suffix .zip or .ZIP)";+ Supported compression formats: gzip (.gz), compress (.Z), zip"; #ifdef HAVE_LIBBZ2 std::cout << ",\n\- bzip2 (requires suffix .bz, .bz2, or .bzip2)";+ bzip2 (requires suffix .bz, .bz2, .bzip2, .tbz, .tbz2, .tb2, .tz2)"; #endif #ifdef HAVE_LIBLZMA std::cout << ",\n\- lzma (requires suffix .lzma), and xz (requires suffix .xz)";+ lzma and xz (requires suffix .lzma, .tlz, .xz, .txz)"; #endif std::cout << ".\n"; #endif
@@ -87,11 +87,12 @@ * to the following four functions, without requiring BSD funopen() and to * avoid a clash with BSD zopen(): *- * z_open(FILE *file, const char *mode, int bits)+ * z_open(FILE *file, const char *mode, int bits, int plain) * Returns a new void* handle to read (decompress) or write (compress) * from/to the specified FILE* with mode "r" or "w" using bits (only used- * for writing, use 0 for default 16 bits), returns NULL when unable to- * allocate.+ * for writing, use 0 for default 16 bits). Normally, plain should be+ * zero, unless the two-byte magic header 1F 9D is, or should be, removed+ * from the compressed stream. Returns NULL when unable to allocate. * * z_read(void *, unsigned char *buf, int len) * Read and decompress up to len bytes into buf, returns the number of@@ -148,6 +149,7 @@ static char_type magic_header[] = struct s_zstate { FILE *zs_fp; /* File stream for I/O */ char zs_mode; /* r or w */+ char zs_magic; /* read or write the magic header */ enum { S_START, S_MIDDLE, S_EOF } zs_state; /* State of computation */@@ -192,6 +194,7 @@ struct s_zstate { /* Definitions to retain old variable names */ #define fp zs->zs_fp #define zmode zs->zs_mode+#define zmagic zs->zs_magic #define state zs->zs_state #define n_bits zs->zs_n_bits #define maxbits zs->zs_maxbits@@ -299,9 +302,12 @@ z_write(void *handle, const unsigned char *wbp, int num) state = S_MIDDLE; maxmaxcode = 1L << maxbits;- if (fwrite(magic_header,- sizeof(char), sizeof(magic_header), fp) != sizeof(magic_header))- return (-1);+ if (zmagic)+ {+ if (fwrite(magic_header,+ sizeof(char), sizeof(magic_header), fp) != sizeof(magic_header))+ return (-1);+ } tmp = (u_char)((maxbits) | block_compress); if (fwrite(&tmp, sizeof(char), sizeof(tmp), fp) != sizeof(tmp)) return (-1);@@ -518,12 +524,12 @@ z_read(void *handle, unsigned char *rbp, int num) /* Check the magic number */ if (fread(header,- sizeof(char), sizeof(header), fp) != sizeof(header) ||- memcmp(header, magic_header, sizeof(magic_header)) != 0) {+ sizeof(char), zmagic + 1, fp) != zmagic + 1 ||+ memcmp(header, magic_header, zmagic) != 0) { errno = EINVAL; return (-1); }- maxbits = header[2]; /* Set -b from file. */+ maxbits = header[zmagic]; /* Set -b from file. */ block_compress = maxbits & BLOCK_MASK; maxbits &= BIT_MASK; maxmaxcode = 1L << maxbits;@@ -731,7 +737,7 @@ cl_hash(struct s_zstate *zs, count_int cl_hsize) /* Reset code table. */ } void *-z_open(FILE *file, const char *mode, int bits)+z_open(FILE *file, const char *mode, int bits, int plain) { struct s_zstate *zs; @@ -756,6 +762,7 @@ z_open(FILE *file, const char *mode, int bits) in_count = 1; /* Length of input. */ out_count = 0; /* # of codes output (for debugging). */ state = S_START;+ zmagic = plain ? 0 : 2; /* plain compressed, without magic header */ roffset = 0; size = 0;
@@ -76,7 +76,7 @@ extern "C" { #endif -void *z_open(FILE *, const char *, int);+void *z_open(FILE *, const char *, int, int); int z_close(void *); int z_read(void *, unsigned char *, int); int z_write(void *, const unsigned char *, int);
@@ -30,7 +30,7 @@ @file zstream.hpp @brief file decompression streams written in C++11 @author Robert van Engelen - engelen@genivia.com-@copyright (c) 2019-2019, Robert van Engelen, Genivia Inc. All rights reserved.+@copyright (c) 2019-2020, Robert van Engelen, Genivia Inc. All rights reserved. @copyright (c) BSD-3 License - see LICENSE.txt */ @@ -63,9 +63,9 @@ struct lzma_stream; // TODO this feature is disabled as it is too slow, we should optimize crc32() with a table // use zip crc integrity check at the cost of a significant slow down?-// #define WITH_CRC32+// #define WITH_ZIP_CRC32 -// buffer size to hold compressed data, e.g. from compressed files+// buffer size to hold compressed data copied from compressed files #ifndef Z_BUF_LEN #define Z_BUF_LEN (65536) #endif@@ -88,7 +88,7 @@ class zstreambuf : public std::streambuf { enum class Compression : uint16_t { STORE = 0, DEFLATE = 8, BZIP2 = 12, LZMA = 14, XZ = 95 }; // constructor- ZipInfo(const char *pathname, FILE *file)+ ZipInfo(const char *pathname, FILE *file, const unsigned char *buf = NULL, size_t len = 0) : pathname_(pathname), file_(file),@@ -100,7 +100,14 @@ class zstreambuf : public std::streambuf { zcrc_(0xffffffff), znew_(true), zend_(false)- { }+ {+ // copy initial buffer data, when specified+ if (buf != NULL && len > 0)+ {+ zlen_ = (len < Z_BUF_LEN ? len : Z_BUF_LEN);+ memcpy(zbuf_, buf, zlen_);+ }+ } // no copy constructor ZipInfo(const ZipInfo&) = delete;@@ -134,8 +141,10 @@ class zstreambuf : public std::streambuf { static constexpr size_t ZIPBLOCK = 65536; // block size to read zip data, at least 64K to fit name - static constexpr uint32_t ZIP_HEADER_MAGIC = 0x04034b50; // zip local file header magic- static constexpr uint32_t ZIP_DESCRIPTOR_MAGIC = 0x08074b50; // zip descriptor magic+ static constexpr uint16_t COMPRESS_HEADER_MAGIC = 0x9d1f; // compress header magic+ static constexpr uint16_t DEFLATE_HEADER_MAGIC = 0x8b1f; // zlib deflate header magic+ static constexpr uint32_t ZIP_HEADER_MAGIC = 0x04034b50; // zip local file header magic+ static constexpr uint32_t ZIP_DESCRIPTOR_MAGIC = 0x08074b50; // zip descriptor magic // convert 2 bytes to 16 bit static uint16_t u16(const unsigned char *buf)@@ -215,7 +224,7 @@ class zstreambuf : public std::streambuf { return false; } - for (uint16_t num = 0; num < extralen; num += u16(data + num + 2))+ for (uint16_t num = 0; num < extralen; num += 4 + u16(data + num + 2)) { uint16_t id = u16(data + num); @@ -269,7 +278,6 @@ class zstreambuf : public std::streambuf { } else if (method == Compression::BZIP2) {- #ifdef HAVE_LIBBZ2 // Zip bzip2 method@@ -309,11 +317,9 @@ class zstreambuf : public std::streambuf { return false; #endif- } else if (method == Compression::LZMA || method == Compression::XZ) {- #ifdef HAVE_LIBLZMA // TODO should we support bit 1 clear, i.e. use the size field to terminate the lzma stream?@@ -358,7 +364,6 @@ class zstreambuf : public std::streambuf { return false; #endif- } else if (method != Compression::STORE || (flag & 8) != 0) {@@ -375,7 +380,7 @@ class zstreambuf : public std::streambuf { return true; } - // read and decompress zip file data+ // read and decompress zip file data into buf[0..len-1], return number of bytes decompressed std::streamsize decompress(unsigned char *buf, size_t len) { // if no more data to decompress, then return 0 to indicate EOF@@ -398,7 +403,7 @@ class zstreambuf : public std::streambuf { zcur_ = zlen_ - z_strm_->avail_in; - if (ret != Z_OK && ret != Z_STREAM_END)+ if (ret != Z_OK && ret != Z_STREAM_END && ret != Z_BUF_ERROR) { cannot_decompress(pathname_, "a zlib decompression error was detected in the zip compressed data"); num = -1;@@ -429,7 +434,7 @@ class zstreambuf : public std::streambuf { zcur_ = zlen_ - z_strm_->avail_in; - if (ret != Z_OK && ret != Z_STREAM_END)+ if (ret != Z_OK && ret != Z_STREAM_END && ret != Z_BUF_ERROR) { cannot_decompress(pathname_, "a zlib decompression error was detected in the zip compressed data"); num = -1;@@ -608,7 +613,7 @@ class zstreambuf : public std::streambuf { #endif else {- // copy the stored zip data until its size is reduced to zero+ // copy the stored zip data until the stored data size is reduced to zero if (size > 0) { if (zcur_ < zlen_ || read())@@ -634,7 +639,7 @@ class zstreambuf : public std::streambuf { zend_ = true; } -#ifdef WITH_CRC32+#ifdef WITH_ZIP_CRC32 // update zip crc32 crc32(buf, static_cast<size_t>(num)); #endif@@ -666,7 +671,7 @@ class zstreambuf : public std::streambuf { // we arrived at a new zip local file header znew_ = true; -#ifdef WITH_CRC32+#ifdef WITH_ZIP_CRC32 // now that we have the crc32 value (from the header or the descriptor), check the integrity of the decompressed data if (~zcrc_ != crc) {@@ -752,7 +757,7 @@ class zstreambuf : public std::streambuf { return zlen_ > 0; } -#ifdef WITH_CRC32+#ifdef WITH_ZIP_CRC32 // update zip crc32 void crc32(const unsigned char *buf, size_t len) {@@ -822,8 +827,9 @@ class zstreambuf : public std::streambuf { zstreambuf(const char *pathname, FILE *file) : pathname_(pathname),+ file_(file),+ zfile_(NULL), zzfile_(NULL),- gzfile_(Z_NULL), bzfile_(NULL), xzfile_(NULL), zipinfo_(NULL),@@ -834,12 +840,22 @@ class zstreambuf : public std::streambuf { { #ifdef HAVE_LIBBZ2 // open bzip2 compressed file- int err = 0;- if ((bzfile_ = BZ2_bzReadOpen(&err, file, 0, 0, NULL, 0)) == NULL || err != BZ_OK)+ try {- warning("BZ2_bzReadOpen error", pathname);+ bzfile_ = new BZ();+ int ret = BZ2_bzDecompressInit(&bzfile_->strm, 0, 0);+ if (ret != BZ_OK)+ {+ warning("BZ2_bzDecompressInit error", pathname); - bzfile_ = NULL;+ delete bzfile_;+ bzfile_ = NULL;+ file_ = NULL;+ }+ }+ catch (const std::bad_alloc&)+ {+ cannot_decompress(pathname_, "out of memory"); } #else cannot_decompress("unsupported compression format", pathname);@@ -851,7 +867,7 @@ class zstreambuf : public std::streambuf { // open xz/lzma compressed file try {- xzfile_ = new XZ(file);+ xzfile_ = new XZ(); lzma_ret ret = lzma_auto_decoder(&xzfile_->strm, UINT64_MAX, LZMA_TELL_UNSUPPORTED_CHECK | LZMA_CONCATENATED); if (ret != LZMA_OK) {@@ -859,6 +875,7 @@ class zstreambuf : public std::streambuf { delete xzfile_; xzfile_ = NULL;+ file_ = NULL; } } catch (const std::bad_alloc&)@@ -869,46 +886,97 @@ class zstreambuf : public std::streambuf { cannot_decompress("unsupported compression format", pathname); #endif }- else if (is_Z(pathname))- {- // open compress (Z) compressed file- if ((zzfile_ = z_open(file, "r", 0)) == NULL)- warning("zopen error", pathname);- }- else if (is_zip(pathname))+ else {- // open zip compressed file- try+ // read compression format magic bytes+ std::streamsize num = fread(buf_, 1, 2, file);++ if (num == 2) {- zipinfo_ = new ZipInfo(pathname, file);+ if (ZipInfo::u16(buf_) == ZipInfo::DEFLATE_HEADER_MAGIC)+ {+ // open zlib compressed file+ try+ {+ zfile_ = new Z();++ // copy the gzip header's magic bytes to zbuf[], needed by inflate()+ zfile_->zbuf[0] = buf_[0];+ zfile_->zbuf[1] = buf_[1];+ zfile_->zlen = 2;+ zfile_->strm.next_in = zfile_->zbuf;+ zfile_->strm.avail_in = zfile_->zlen; - // read the zip header of the first compressed file- if (!zipinfo_->header())+ // inflate gzip compressed data starting with a gzip header+ if (inflateInit2(&zfile_->strm, 16 + MAX_WBITS) != Z_OK)+ {+ cannot_decompress(pathname_, zfile_->strm.msg ? zfile_->strm.msg : "inflateInit2 failed");++ delete zfile_;+ zfile_ = NULL;+ file_ = NULL;+ }+ }+ catch (const std::bad_alloc&)+ {+ errno = ENOMEM;+ warning("zlib open error", pathname);+ }+ }+ else if (ZipInfo::u16(buf_) == ZipInfo::COMPRESS_HEADER_MAGIC) {- delete zipinfo_;- zipinfo_ = NULL;+ // open compress (Z) compressed file+ if ((zzfile_ = z_open(file, "r", 0, 1)) == NULL)+ warning("zopen error", pathname);+ }+ else+ {+ // read two more bytes of the compression format's magic bytes+ num = fread(buf_ + 2, 1, 2, file);++ if (num == 2 && ZipInfo::u32(buf_) == ZipInfo::ZIP_HEADER_MAGIC)+ {+ // open zip compressed file+ try+ {+ zipinfo_ = new ZipInfo(pathname, file, buf_, 4);++ // read the zip header of the first compressed file+ if (!zipinfo_->header())+ {+ delete zipinfo_;+ zipinfo_ = NULL;+ file_ = NULL;+ }+ }+ catch (const std::bad_alloc&)+ {+ errno = ENOMEM;+ warning("zip open error", pathname);+ }+ }+ else if (num >= 0)+ {+ // no compression+ len_ = num + 2;+ num = fread(buf_ + 4, 1, Z_BUF_LEN - 4, file);+ if (num >= 0)+ {+ len_ += num;+ }+ else+ {+ warning("cannot read", pathname);+ file_ = NULL;+ }+ } } }- catch (const std::bad_alloc&)- {- errno = ENOMEM;- warning("zip open error", pathname);- }- }- else- {- // open gzip compressed file- int fd = dup(fileno(file));- if (fd >= 0 && (gzfile_ = gzdopen(fd, "r")) != Z_NULL)- {- gzbuffer(gzfile_, Z_BUF_LEN);- }- else- {- warning("gzdopen error", pathname); - if (fd >= 0)- close(fd);+ if (num < 0)+ {+ warning("cannot read", pathname);+ file_ = NULL; } } }@@ -919,77 +987,406 @@ class zstreambuf : public std::streambuf { // destructor virtual ~zstreambuf() {+ if (zfile_ != NULL)+ {+ // close zlib compressed file+ delete zfile_;+ }+ else if (zzfile_ != NULL)+ {+ // close compress (Z) compressed file+ z_close(zzfile_);+ } #ifdef HAVE_LIBBZ2- if (bzfile_ != NULL)+ else if (bzfile_ != NULL) { // close bzlib compressed file- int err = 0;- BZ2_bzReadClose(&err, bzfile_);+ delete bzfile_; }- else #endif #ifdef HAVE_LIBLZMA- if (xzfile_ != NULL)+ else if (xzfile_ != NULL) { // close lzma compressed file delete xzfile_; } else #endif- if (zzfile_ != NULL)- {- // close compress (Z) compressed file- z_close(zzfile_);- }- else if (zipinfo_ != NULL)+ if (zipinfo_ != NULL) { // close zip compressed file delete zipinfo_; }- else if (gzfile_ != Z_NULL)+ }++ // copy or decompress a block of data into buf[0..len-1], return number of bytes decompressed, zero on EOF or negative on error+ std::streamsize decompress(unsigned char *buf, size_t len)+ {+ if (cur_ >= len_)+ return next(buf, len);++ size_t num = static_cast<size_t>(len_ - cur_);+ if (num > len)+ num = len;++ memcpy(buf, buf_ + cur_, num);++ cur_ += num;++ return static_cast<std::streamsize>(num);+ }++ // get pointer to the internal buffer and its max size+ void get_buffer(unsigned char *& buffer, size_t& maxlen)+ {+ buffer = buf_;+ maxlen = Z_BUF_LEN;+ }++ // return zip info when unzipping a file, if at end then proceed to the next file to unzip, otherwise return NULL, + const ZipInfo *zipinfo()+ {+ if (zipinfo_ && !zipinfo_->header()) {- // close zlib compressed file- gzclose_r(gzfile_);+ delete zipinfo_;+ zipinfo_ = NULL; }++ return zipinfo_; } - // decompress a block of data into buf[0..len-1], return number of bytes decompressed, zero on stream end or negative on error- std::streamsize decompress(unsigned char *buf, size_t len)+ // return pointer and length to current data when unzipping a file, NULL otherwise+ std::pair<const unsigned char*,size_t> zippeek() {- std::streamsize num = 0;+ if (zipinfo_)+ return zipinfo_->peek();++ return std::pair<const unsigned char*,size_t>(NULL, 0);+ }++ protected:++ // zlib deflate data+ struct Z {++ Z()+ :+ zlen(0),+ zend(false)+ {+ strm.zalloc = Z_NULL;+ strm.zfree = Z_NULL;+ strm.opaque = Z_NULL;+ strm.next_in = Z_NULL;+ strm.avail_in = 0;+ strm.next_out = Z_NULL;+ strm.avail_out = 0;+ }++ ~Z()+ {+ inflateEnd(&strm);+ }++ z_stream strm;+ unsigned char zbuf[Z_BUF_LEN];+ size_t zlen;+ bool zend;++ };+ + // zlib deflate file handle+ typedef struct Z *zFile;++ // compress (Z) file handle+ typedef void *zzFile; #ifdef HAVE_LIBBZ2- if (bzfile_ != NULL)++ // bzip2 state data+ struct BZ {++ BZ()+ :+ zlen(0),+ zend(false) {- // decompress a bzlib compressed block into the given buf[]- int err = 0;- num = BZ2_bzRead(&err, bzfile_, buf, len);+ strm.bzalloc = NULL;+ strm.bzfree = NULL;+ strm.opaque = NULL;+ strm.next_in = NULL;+ strm.avail_in = 0;+ strm.next_out = NULL;+ strm.avail_out = 0;+ } - // an error occurred?- if (err != BZ_OK && err != BZ_STREAM_END)+ ~BZ()+ {+ BZ2_bzDecompressEnd(&strm);+ }++ bz_stream strm;+ unsigned char zbuf[Z_BUF_LEN];+ size_t zlen;+ bool zend;++ };+ + // bzip2 file handle+ typedef struct BZ *bzFile;++#else++ // unused bzip2 file handle+ typedef void *bzFile;++#endif++#ifdef HAVE_LIBLZMA++ // lzma and xz state data+ struct XZ {++ XZ()+ :+ strm(LZMA_STREAM_INIT),+ zlen(0),+ zend(false)+ { }++ ~XZ()+ {+ lzma_end(&strm);+ }++ lzma_stream strm;+ unsigned char zbuf[Z_BUF_LEN];+ size_t zlen;+ bool zend;++ };+ + // lzma/xz file handle+ typedef struct XZ *xzFile;++#else++ // unused lzma/xz file handle+ typedef void *xzFile;++#endif++ // fetch and decompress the next block of data into buf[0..len-1], return number of bytes decompressed, zero on EOF or negative on error+ std::streamsize next(unsigned char *buf, size_t len)+ {+ std::streamsize num = 0;++ if (zfile_ != NULL)+ {+ while (true) {- cannot_decompress(pathname_, "an error was detected in the bzip2 compressed data");- num = -1;+ int ret = Z_OK;++ // decompress non-empty zfile_->zbuf[] into the given buf[]+ if (zfile_->strm.avail_in > 0 && zfile_->strm.avail_out == 0)+ {+ zfile_->strm.next_out = buf;+ zfile_->strm.avail_out = len;++ ret = inflate(&zfile_->strm, Z_NO_FLUSH);++ if (ret != Z_OK && ret != Z_STREAM_END && ret != Z_BUF_ERROR)+ {+ cannot_decompress(pathname_, "an error was detected in the gzip compressed data");+ num = -1;+ }+ else+ {+ num = len - zfile_->strm.avail_out;+ }+ }++ // read compressed data into zfile_->zbuf[] and decompress zfile_->zbuf[] into the given buf[]+ if (ret == Z_OK && zfile_->strm.avail_in == 0 && num < static_cast<std::streamsize>(len) && !zfile_->zend)+ {+ zfile_->zlen = fread(zfile_->zbuf, 1, Z_BUF_LEN, file_);++ if (ferror(file_))+ {+ warning("cannot read", pathname_);+ zfile_->zend = true;+ }+ else+ {+ if (feof(file_))+ zfile_->zend = true;++ zfile_->strm.next_in = zfile_->zbuf;+ zfile_->strm.avail_in = zfile_->zlen;++ if (num == 0)+ {+ zfile_->strm.next_out = buf;+ zfile_->strm.avail_out = len;+ }++ ret = inflate(&zfile_->strm, Z_NO_FLUSH);++ if (ret != Z_OK && ret != Z_STREAM_END && ret != Z_BUF_ERROR)+ {+ cannot_decompress(pathname_, "an error was detected in the gzip compressed data");+ num = -1;+ }+ else+ {+ num = len - zfile_->strm.avail_out;+ }+ }+ }++ if (num >= 0 && ret == Z_STREAM_END)+ {+ // try to decompress the next concatenated gzip data+ if (inflateReset(&zfile_->strm) == Z_OK)+ {+ zfile_->strm.next_out = Z_NULL;+ zfile_->strm.avail_out = 0;++ ret = Z_OK;++ if (num == 0)+ continue;+ }+ }++ // decompressed the last block or there was an error?+ if (num <= 0 || ret == Z_STREAM_END)+ {+ delete zfile_;+ zfile_ = NULL;+ file_ = NULL;+ }++ break; }+ }+ else if (zzfile_ != NULL)+ {+ // decompress a compress (Z) compressed block into the given buf[]+ num = z_read(zzfile_, buf, static_cast<int>(len)); ++ // an error occurred?+ if (num < 0)+ cannot_decompress(pathname_, "an error was detected in the compressed data"); // decompressed the last block or there was an error?- if (num <= 0 || err == BZ_STREAM_END)+ if (num <= 0) {- BZ2_bzReadClose(&err, bzfile_);- bzfile_ = NULL;+ z_close(zzfile_);+ zzfile_ = NULL;+ file_ = NULL;+ }+ }+#ifdef HAVE_LIBBZ2+ else if (bzfile_ != NULL)+ {+ while (true)+ {+ int ret = BZ_OK;++ // decompress non-empty bzfile_->zbuf[] into the given buf[]+ if (bzfile_->strm.avail_in > 0 && bzfile_->strm.avail_out == 0)+ {+ bzfile_->strm.next_out = reinterpret_cast<char*>(buf);+ bzfile_->strm.avail_out = len;++ ret = BZ2_bzDecompress(&bzfile_->strm);++ if (ret != BZ_OK && ret != BZ_STREAM_END)+ {+ cannot_decompress(pathname_, "an error was detected in the bzip2 compressed data");+ num = -1;+ }+ else+ {+ num = len - bzfile_->strm.avail_out;+ }+ }++ // read compressed data into bzfile_->zbuf[] and decompress bzfile_->zbuf[] into the given buf[]+ if (ret == BZ_OK && bzfile_->strm.avail_in == 0 && num < static_cast<std::streamsize>(len) && !bzfile_->zend)+ {+ bzfile_->zlen = fread(bzfile_->zbuf, 1, Z_BUF_LEN, file_);++ if (ferror(file_))+ {+ warning("cannot read", pathname_);+ bzfile_->zend = true;+ }+ else+ {+ if (feof(file_))+ bzfile_->zend = true;++ bzfile_->strm.next_in = reinterpret_cast<char*>(bzfile_->zbuf);+ bzfile_->strm.avail_in = bzfile_->zlen;++ if (num == 0)+ {+ bzfile_->strm.next_out = reinterpret_cast<char*>(buf);+ bzfile_->strm.avail_out = len;+ }++ ret = BZ2_bzDecompress(&bzfile_->strm);++ if (ret != BZ_OK && ret != BZ_STREAM_END)+ {+ cannot_decompress(pathname_, "an error was detected in the bzip2 compressed data");+ num = -1;+ }+ else+ {+ num = len - bzfile_->strm.avail_out;+ }+ }+ }++ if (num >= 0 && ret == BZ_STREAM_END)+ {+ // try to decompress the next concatenated bzip2 data+ if (BZ2_bzDecompressEnd(&bzfile_->strm) == BZ_OK && BZ2_bzDecompressInit(&bzfile_->strm, 0, 0) == BZ_OK)+ {+ bzfile_->strm.next_out = NULL;+ bzfile_->strm.avail_out = 0;++ ret = BZ_OK;++ if (num == 0)+ continue;+ }+ }++ // decompressed the last block or there was an error?+ if (num <= 0 || ret == BZ_STREAM_END)+ {+ delete bzfile_;+ bzfile_ = NULL;+ file_ = NULL;+ }++ break; } }- else #endif #ifdef HAVE_LIBLZMA- if (xzfile_ != NULL)+ else if (xzfile_ != NULL) { lzma_ret ret = LZMA_OK; // decompress non-empty xzfile_->zbuf[] into the given buf[] if (xzfile_->strm.avail_in > 0 && xzfile_->strm.avail_out == 0) {- xzfile_->strm.next_out = buf;+ xzfile_->strm.next_out = buf; xzfile_->strm.avail_out = len; ret = lzma_code(&xzfile_->strm, xzfile_->zend ? LZMA_FINISH : LZMA_RUN);@@ -1008,24 +1405,24 @@ class zstreambuf : public std::streambuf { // read compressed data into xzfile_->zbuf[] and decompress xzfile_->zbuf[] into the given buf[] if (ret == LZMA_OK && xzfile_->strm.avail_in == 0 && num < static_cast<std::streamsize>(len) && !xzfile_->zend) {- xzfile_->zlen = fread(xzfile_->zbuf, 1, Z_BUF_LEN, xzfile_->file);+ xzfile_->zlen = fread(xzfile_->zbuf, 1, Z_BUF_LEN, file_); - if (ferror(xzfile_->file))+ if (ferror(file_)) { warning("cannot read", pathname_); xzfile_->zend = true; } else {- if (feof(xzfile_->file))+ if (feof(file_)) xzfile_->zend = true; - xzfile_->strm.next_in = xzfile_->zbuf;+ xzfile_->strm.next_in = xzfile_->zbuf; xzfile_->strm.avail_in = xzfile_->zlen; if (num == 0) {- xzfile_->strm.next_out = buf;+ xzfile_->strm.next_out = buf; xzfile_->strm.avail_out = len; } @@ -1048,26 +1445,10 @@ class zstreambuf : public std::streambuf { { delete xzfile_; xzfile_ = NULL;+ file_ = NULL; } }- else #endif- if (zzfile_ != NULL)- {- // decompress a compress (Z) compressed block into the given buf[]- num = z_read(zzfile_, buf, static_cast<int>(len)); -- // an error occurred?- if (num < 0)- cannot_decompress(pathname_, "an error was detected in the compressed data");-- // decompressed the last block or there was an error?- if (num <= 0)- {- z_close(zzfile_);- zzfile_ = NULL;- }- } else if (zipinfo_ != NULL) { // decompress a zip compressed block into the guven buf[]@@ -1078,113 +1459,27 @@ class zstreambuf : public std::streambuf { { delete zipinfo_; zipinfo_ = NULL;+ file_ = NULL; } }- else if (gzfile_ != Z_NULL)+ else if (file_ != NULL) {- // decompress a zlib compressed block into the given buf[]- num = gzread(gzfile_, buf, len);-- // an error occurred?- if (num < 0)- {- int err;- const char *message = gzerror(gzfile_, &err);+ // pass through, without decompression+ num = fread(buf, 1, len, file_); - if (err == Z_ERRNO)- warning("cannot read", pathname_);- else- cannot_decompress(pathname_, message);- }-- // decompressed the last block?+ // end of file or error? if (num < static_cast<std::streamsize>(len))- {- gzclose_r(gzfile_);- gzfile_ = Z_NULL;- }+ file_ = NULL; } return num; } - // get pointer to the internal buffer and its max size- void get_buffer(unsigned char *& buffer, size_t& maxlen)- {- buffer = buf_;- maxlen = Z_BUF_LEN;- }-- // return zip info when unzipping a file, if at end then proceed to the next file to unzip, otherwise return NULL, - const ZipInfo *zipinfo()- {- if (zipinfo_ && !zipinfo_->header())- {- delete zipinfo_;- zipinfo_ = NULL;- }-- return zipinfo_;- }-- // return pointer and length to current data when unzipping a file, NULL otherwise- std::pair<const unsigned char*,size_t> zippeek()- {- if (zipinfo_)- return zipinfo_->peek();-- return std::pair<const unsigned char*,size_t>(NULL, 0);- }-- protected:-- // compress (Z) file handle- typedef void *zzFile;-- // bzip2 file handle- typedef void *bzFile;--#ifdef HAVE_LIBLZMA-- // lzma and xz state data- struct XZ {-- XZ(FILE *file)- :- file(file),- strm(LZMA_STREAM_INIT),- zlen(0),- zend(false)- { }-- ~XZ()- {- lzma_end(&strm);- }-- FILE *file;- lzma_stream strm;- unsigned char zbuf[Z_BUF_LEN];- size_t zlen;- bool zend;-- };- - // lzma/xz file handle- typedef struct XZ *xzFile;--#else-- // unused lzma file handle- typedef void *xzFile;--#endif- // read a decompressed block into buf_[], returns pending next character or EOF int_type peek() { cur_ = 0;- len_ = decompress(buf_, Z_BUF_LEN);+ len_ = next(buf_, Z_BUF_LEN); return len_ > 0 ? traits_type::to_int_type(*buf_) : traits_type::eof(); } @@ -1212,7 +1507,7 @@ class zstreambuf : public std::streambuf { // std::streambuf::showmanyc() std::streamsize showmanyc() override {- return gzfile_ == Z_NULL && bzfile_ == NULL && cur_ >= len_ ? -1 : 0;+ return file_ == NULL && cur_ >= len_ ? -1 : 0; } // std::streambuf::xsgetn(s, n)@@ -1239,11 +1534,12 @@ class zstreambuf : public std::streambuf { } const char *pathname_; // the pathname of the compressed file+ FILE *file_; // the compressed file+ zFile zfile_; // zlib file handle zzFile zzfile_; // compress (Z) file handle- gzFile gzfile_; // zlib file handle bzFile bzfile_; // bzip2 file handle xzFile xzfile_; // lzma/xz file handle- ZipInfo *zipinfo_; // zip file and info handle+ ZipInfo *zipinfo_; // zip file and zip info handle unsigned char buf_[Z_BUF_LEN]; // buffer with decompressed stream data std::streamsize cur_; // current position in buffer to read the stream data, less or equal to len_ std::streamsize len_; // length of decompressed data in the buffer