191203.1 · 974ff24fb56f1a6eb7bc44028f6320a170e78fad
released 1.6.4 - windows usability improvements, added HTML patterns
reading declarations
@@ -104,7 +104,8 @@ Introduction: why use ugrep? looks for `main` in C++ files in the compressed tar file `myprojects.tgz`. Supported tar formats are v7, ustar, gnu, oldgnu, and pax. Supported cpio- formats are odc and newc (not the non-portable old binary cpio format).+ formats are odc, newc, and crc (not the obsolete non-portable old binary cpio+ format). - **ugrep can match patterns across multiple lines**, such as comment blocks in source code. This feature supports matching that could otherwise only be@@ -196,16 +197,16 @@ Introduction: why use ugrep? - **ugrep is evolving!** We added lots of useful features and made significant improvements. This effort continues. You can help! We love your feedback (issues) and contributions (pull requests) ❤️ For example, what patterns do- you use to search source code? Please contribute and share!+ you use to search source code? Please tell us or contribute and share! <a name="speed"/> Speed ----- -Performance tests were conducted with clang 9.0.0 -O2 on on a 2.9 GHz Intel-Core i7, 16 GB 2133 MHz LPDDR3 Mac OS 10.12.6 machine. The best times for many-runs is shown under minimal machine load.+Performance tests were conducted with clang 9.0.0 -O2 on a 2.9 GHz Intel Core+i7, 16 GB 2133 MHz LPDDR3 Mac OS 10.12.6 machine. The best times for many runs+is shown under minimal machine load. The following results span a range of different tests and are promosing for our initial efforts on **ugrep**. These tests are not comprehensive (yet), but@@ -358,8 +359,19 @@ Prebuilt binaries for Linux, Mac OS X, and Windows are included in the `bin` directory. These binary versions do not support options `-P` (Perl regular expressions) and `-z` (decompress). +All versions of ugrep are designed to run from the command line interface+(CLI).++There are two Windows versions: `ugrep\bin\win32\ugrep.exe` and+`ugrep\bin\win64\ugrep.exe`. Depending on your system, add the 32 or 64 bit+version to your execution path: go to *Settings* and search for "Path" in+*Find a Setting*. Select *environment variables* -> *Path* -> *New* and add+the directory `C:\<fill this part in>\ugrep\bin\win32`.+ When using `ugrep.exe` from the Windows command line, use `"` instead of `'` to-specify patterns, since `'` becomes part of the command-line argument!+specify patterns, since `'` becomes part of the command-line argument! Also,+an empty argument `""` is ignored by some Windows command interpreters such as+Powershell. Use option `--match` instead. The Linux binary was built on CentOS 7.6. @@ -370,7 +382,7 @@ The Linux binary was built on CentOS 7.6. Add to `.vimrc`: if executable('ugrep')- set grepprg=ugrep\ -RHnk\ -u\ --tabs=1+ set grepprg=ugrep\ -Rnk\ -u\ --tabs=1 set grepformat=%f:%l:%c:%m,%f+%l+%c+%m,%-G%f\\\|%l\\\|%c\\\|%m endif @@ -786,17 +798,22 @@ Note that regex patterns are always specified in UTF-8 (includes ASCII). To search binary files with binary patterns, see [searching and displaying binary files with -U, -W, and -X](#binary). -To recursively list all files and symlinks that are ASCII:+To recursively list all files and symlinks that are ASCII (i.e. 7-bit): - ugrep -RL '[\p{Non_ASCII_Unicode}]'+ ugrep -RL '[^[:ascii:]]' -To recursively list all files and symlinks that are non-ASCII:+To recursively list all files and symlinks that are non-ASCII, i.e. UTF-8,+UTF-16, and UTF-32 files with non-ASCII Unicode characters (U+0080 and up): ugrep -Rl '[^[:ascii:]]' +To check that a file contains Unicode ((U+0080 and up):++ ugrep -q '[^[:ascii:]]' myfile && echo "contains Unicode"+ To recursively list files with invalid UTF content (i.e. invalid UTF-8 byte-sequences or that contain any UTF-8/16/32 code points that are outside the-valid Unicode range) by using a negative pattern:+sequences or files that contain any UTF-8/16/32 code points that are outside+the valid Unicode range) by using a negative pattern: ugrep -Rl '.|(?^\p{Unicode})' @@ -818,14 +835,10 @@ with a UTF-16 BOM: ugrep -iw 'lorem' utf16lorem.txt -To search utf16lorem.txt when this file has no UTF-16 BOM:+To search utf16lorem.txt when this file has no UTF-16 BOM, using `-Q`: ugrep -Q=UTF-16 -iw 'lorem' utf16lorem.txt -To check that a file contains Unicode:-- ugrep -q '[^[:ascii:]]' myfile && echo "contains Unicode"- To search file `spanish-iso.txt` encoded in ISO-8859-1: ugrep -Q=ISO-8859-1 -w 'año' spanish-iso.txt@@ -1136,7 +1149,7 @@ To display the line and column numbers of matches in XML with `--xml`: To change the color palette, set the `GREP_COLORS` environment variable. Its value is a colon-separated list of ANSI SGR parameters that defaults to-`cx=2:mt=1;31:fn=35:ln=32:cn=32:bn=32:se=36`:+`cx=2:mt=1;31:fn=35:ln=32:cn=32:bn=32:se=36` when not assigned: param | result ----- | ------------------------------------------------------------------------@@ -1159,7 +1172,12 @@ lines with a dark gray background, thereby aiding in visualizing white space: export GREP_COLORS='sl=1;100:cx=44:ms=1;4;32;100:mc=1;4;32;44:fn=35:ln=32:cn=32:bn=32:se=36' -Color intensities may differ per platform and terminal used though, which+For Windows command interpreters that support ANSI escape codes, use+`SET GREP_COLORS=<params>`. For example:++ SET GREP_COLORS=cx=2:mt=1;31:fn=35:ln=32:cn=32:bn=32:se=36++Color intensities may differ per platform and per terminal program used, which affects readability. To produce color-highlighted results:@@ -1338,15 +1356,19 @@ The same as above, but displaying the replaced matches line-by-line: ugrep -P '<td>(.*?)</td>' --format='%1\n' index.html -To collect all `<link>` `href` URLs from all HTML files down the current-working directory, then sort them:+To collect all `href` URLs from all HTML and PHP files down the current working+directory, then sort them: - ugrep -R -thtml -P '<link.*href\h*=\h*.([^\x27"\n]+).' --format='%1%~' | sort -u+ ugrep -R -thtml,php -P '<[^<>\n]+href\h*=\h*.([^\x27"\n]+).' --format='%1%~' | sort -u -Likewise, but select `<script>` `src` URLs when referencing `http` and `https`-sites:+The same, but much easier by using the predefined `html/href` pattern: - ugrep -R -thtml -P '<script.*src\h*=\h*.(https?:[^\x27"\n]+).' --format='%1%~' | sort -u+ ugrep -R -thtml,php -P -f html/href --format='%1%~' | sort -u++Likewise, but in this case select `<script>` `src` URLs when referencing `http`+and `https` sites:++ ugrep -R -thtml,php -P '<script.*src\h*=\h*.(https?:[^\x27"\n]+).' --format='%1%~' | sort -u <a name="max"/> @@ -1386,13 +1408,13 @@ use option `--label='stdin.Z'` for compress, `--label='stdin.bz2'` for bzip2, Archives (cpio, tar, and pax) are searched with option `-z`. Regular files in an archive that match are output with the archive pathnames enclosed in `{` and `}` braces. Supported tar formats are v7, ustar, gnu, oldgnu, and pax.-Supported cpio formats are odc and newc. Not supported is the non-portable old-binary cpio format. Archive formats are automatically detected, including-compressed with gzip. Other compression formats require a filename suffix,-`.Z` for compress, `.bz`, `.bz2`, or `.bzip2` for bzip2, `.lzma` for lzma, and-`.xz` for xz. Also the shorthands `.taz`, `.tgz`, and `.tpz` for gzip, `.tbz`,-`.tbz2`, `.tb2`, and `.tz2` for bzip, `.tlz` for lzma, and `.txz` for xz are-recognized.+Supported cpio formats are odc, newc, and crc. Not supported is the obsolete+non-portable old binary cpio format. Archive formats are automatically+detected, including compressed with gzip. Other compression formats require a+filename suffix: `.Z` for compress, `.bz`, `.bz2`, or `.bzip2` for bzip2,+`.lzma` for lzma, and `.xz` for xz. Also the shorthands `.taz`, `.tgz`, and+`.tpz` for gzip, `.tbz`, `.tbz2`, `.tb2`, and `.tz2` for bzip, `.tlz` for lzma,+and `.txz` for xz are recognized. format | filename suffix | tar/pax short suffix | suffix required? | to ugrep stdin | lib install | -------- | ----------------------- | ------------------------------- | ---------------- | ----------------- | ----------- |@@ -1804,6 +1826,9 @@ Man page Stop reading the input after NUM matches for each file pro- cessed. + --match+ Match input. Same as specifying an empty pattern to search.+ --max-depth=NUM Restrict recursive search to NUM (NUM > 0) directories deep, where --max-depth=1 searches the specified path without visiting@@ -2350,7 +2375,7 @@ Man page - ugrep 1.6.3 December 01, 2019 UGREP(1)+ ugrep 1.6.4 December 03, 2019 UGREP(1) <a name="patterns"/>
bin/linux/ugrep
bin/macosx/ugrep
@@ -1,4 +1,4 @@-.TH UGREP "1" "December 01, 2019" "ugrep 1.6.3" "User Commands"+.TH UGREP "1" "December 03, 2019" "ugrep 1.6.4" "User Commands" .SH NAME \fBugrep\fR -- universal file pattern searcher .SH SYNOPSIS@@ -268,6 +268,9 @@ slow as every file on the search path is read to compare MAGIC. \fB\-m\fR \fINUM\fR, \fB\-\-max\-count\fR=\fINUM\fR Stop reading the input after NUM matches for each file processed. .TP+\fB\-\-match\fR+Match input. Same as specifying an empty pattern to search.+.TP \fB\-\-max\-depth\fR=\fINUM\fR Restrict recursive search to NUM (NUM > 0) directories deep, where \fB\-\-max\-depth\fR=1 searches the specified path without visiting
@@ -0,0 +1,13 @@+HTML patterns+=============++- `comments` matches comments+- `href` matches and captures tags with references, i.e. `<sometag...href='(...)'`+- `img` matches `<img>` tags with content+- `link` matches `<link>` tags with content+- `meta` matches `<meta>` tags with content+- `pre` matches preformatted text, i.e. `<pre>...</pre>`+- `script` matches inline and external Javascripts, i.e. `<script>...</script>`+- `src` matches and captures tags with sources, i.e. `<sometag...src='(...)'`+- `style` matches inline CSS, i.e. `<style>...</style>`+- `zap_comments` removes comments from matches
@@ -0,0 +1 @@+<!--([^-]|-[^-]|--+[^->])*-*-->
@@ -0,0 +1 @@+<[^<>\n]+\hhref\h*=\h*['"]([^'"\n]*)['"]
@@ -0,0 +1,2 @@+<img(\h[^<>\n]*)?/?>+
@@ -0,0 +1 @@+<link(\h[^<>\n]*)?/?>
@@ -0,0 +1 @@+<meta(\h[^<>\n]*)?/?>
@@ -0,0 +1 @@+<pre(\h[^<>\n]*)?>(.|\n)*?</pre>
@@ -0,0 +1,2 @@+<script(\h[^<>\n]*)?/>+<script(\h[^<>\n]*)?>(.|\n)*?</script>
@@ -0,0 +1 @@+<[^<>\n]+\hsrc\h*=\h*['"]([^'"\n]*)['"]
@@ -0,0 +1,2 @@+<style(\h[^<>\n]*)?/>+<style(\h[^<>\n]*)?>(.|\n)*?</style>
@@ -110,14 +110,14 @@ Prebuilt executables are located in ugrep/bin. #include <windows.h> #include <tchar.h> #include <stdio.h>+#include <io.h> #include <strsafe.h> #define STDIN_FILENO 0 #define STDOUT_FILENO 1 #define STDERR_FILENO 2 -// windows has no isatty()-#define isatty(fildes) ((fildes) == 1 || (fildes) == 2)+#define isatty(fildes) _isatty(fildes) #define PATHSEPCHR '\\' #define PATHSEPSTR "\\"@@ -149,7 +149,7 @@ Prebuilt executables are located in ugrep/bin. #endif // ugrep version info-#define UGREP_VERSION "1.6.3"+#define UGREP_VERSION "1.6.4" // ugrep platform -- see configure.ac #if !defined(PLATFORM)@@ -248,7 +248,7 @@ bool flag_no_filename = false; bool flag_no_header = false; bool flag_no_messages = false; bool flag_no_hidden = false;-bool flag_match_all = false;+bool flag_match = false; bool flag_count = false; bool flag_fixed_strings = false; bool flag_free_space = false;@@ -1066,6 +1066,8 @@ void Output::Dump::line(const char *separator) { out.str(color_cx); out.str(" --");+ if ((i & 7) == 7)+ out.chr(' '); out.str(color_off); } else@@ -1074,11 +1076,10 @@ void Output::Dump::line(const char *separator) out.str(color_hex[byte >> 8]); out.chr(' '); out.hex(byte & 0xff, 2);+ if ((i & 7) == 7)+ out.chr(' '); out.str(color_off); }-- if ((i & 7) == 7)- out.chr(' '); } out.chr(' ');@@ -3275,6 +3276,8 @@ int main(int argc, char **argv) flag_line_number = true; else if (strcmp(arg, "line-regexp") == 0) flag_line_regexp = true;+ else if (strcmp(arg, "match") == 0)+ flag_match = true; else if (strncmp(arg, "max-count=", 10) == 0) flag_max_count = strtopos(arg + 10, "invalid argument --max-count="); else if (strncmp(arg, "max-depth=", 10) == 0)@@ -3684,10 +3687,15 @@ int main(int argc, char **argv) } } +#ifndef OS_WIN+ // ignore SIGPIPE+ signal(SIGPIPE, SIG_IGN);+#endif+ #ifndef HAVE_LIBZ // -z: but we don't have libz if (flag_decompress)- help("option -z is not available in this version of ugrep");+ help("option -z is not available in this build configuration of ugrep"); #endif // --binary-files: normalize by assigning flags@@ -3719,12 +3727,16 @@ int main(int argc, char **argv) exit(EXIT_ERROR); } + // --match-all is the same as specifying an '' empty pattern argument+ if (flag_match)+ flag_regexp.emplace_back("");+ // regex PATTERN specified if (pattern != NULL) {- // if one or more -e PATTERN given, add pattern to the front else add to the front of FILE args+ // if not --match-all, then add pattern to the front else add to the front of FILE args if (flag_regexp.empty())- flag_regexp.insert(flag_regexp.begin(), pattern);+ flag_regexp.emplace_back(pattern); else files.insert(files.begin(), pattern); }@@ -3753,12 +3765,12 @@ int main(int argc, char **argv) else if (flag_hex) { regex.append(".*\n?|");- flag_match_all = true;+ flag_match = true; } else { regex.append(".*|");- flag_match_all = true;+ flag_match = true; } flag_empty = true;@@ -3853,11 +3865,17 @@ int main(int argc, char **argv) else if (fopen_s(&file, filename.c_str(), "r") != 0) file = NULL; -#ifndef OS_WIN if (file == NULL) {+#ifdef OS_WIN+ // could not open, try GREP_PATH environment variable+ char *grep_path;+ size_t len;+ _dupenv_s(&grep_path, &len, "GREP_PATH");+#else // could not open, try GREP_PATH environment variable const char *grep_path = getenv("GREP_PATH");+#endif if (grep_path != NULL) {@@ -3866,9 +3884,12 @@ int main(int argc, char **argv) if (fopen_s(&file, path_file.c_str(), "r") != 0) file = NULL;++#ifdef OS_WIN+ free(grep_path);+#endif } }-#endif #ifdef GREP_PATH if (file == NULL)@@ -4005,13 +4026,10 @@ int main(int argc, char **argv) // is output sent to a TTY or to /dev/null? if (!flag_quiet) {-#ifndef OS_WIN-- // ignore SIGPIPE- signal(SIGPIPE, SIG_IGN);- // check if standard output is a TTY- tty_term = isatty(STDOUT_FILENO);+ tty_term = isatty(STDOUT_FILENO) != 0;++#ifndef OS_WIN // --pager: if output is to a TTY then page through the results if (flag_pager != NULL && tty_term)@@ -4054,7 +4072,12 @@ int main(int argc, char **argv) } else {-#ifndef OS_WIN+#ifdef OS_WIN++ // blindly assume that we have a color terminal on Windows if isatty() is true+ color_term = tty_term;++#else // check whether we have a color terminal if (tty_term)@@ -4081,11 +4104,19 @@ int main(int argc, char **argv) if (flag_color != NULL) {+#ifdef OS_WIN+ // get GREP_COLOR and GREP_COLORS+ char *grep_color = NULL;+ char *grep_colors = NULL;++ size_t len;+ _dupenv_s(&grep_color, &len, "GREP_COLOR");+ _dupenv_s(&grep_colors, &len, "GREP_COLORS");+#else+ // get GREP_COLOR and GREP_COLORS environment variables const char *grep_color = NULL; const char *grep_colors = NULL; -#ifndef OS_WIN- // get GREP_COLOR and GREP_COLORS environment variables grep_color = getenv("GREP_COLOR"); grep_colors = getenv("GREP_COLORS"); #endif@@ -4119,7 +4150,7 @@ int main(int argc, char **argv) } // if pattern is empty to match all, matches are shown as selected lines- if (flag_match_all)+ if (flag_match) { copy_color(color_ms, color_sl); copy_color(color_mc, color_cx);@@ -4137,6 +4168,13 @@ int main(int argc, char **argv) color_off = "\033[0m"; }++#ifdef OS_WIN+ if (grep_color != NULL)+ free(grep_color);+ if (grep_colors != NULL)+ free(grep_colors);+#endif } } }@@ -4557,7 +4595,7 @@ int main(int argc, char **argv) ugrep(magic, grep, files); } #else- help("option -P is not available in this version of ugrep");+ help("option -P is not available in this build configuration of ugrep"); #endif } else@@ -4735,7 +4773,14 @@ void find(size_t level, reflex::Matcher& magic, Grep& grep, const char *pathname DWORD attr = GetFileAttributesA(pathname); - if (flag_no_hidden && (attr & FILE_ATTRIBUTE_HIDDEN))+ if (attr == INVALID_FILE_ATTRIBUTES)+ {+ errno = ENOENT;+ warning("cannot read", pathname);+ return;+ }++ if (flag_no_hidden && ((attr & FILE_ATTRIBUTE_HIDDEN) || (attr & FILE_ATTRIBUTE_SYSTEM))) return; if ((attr & FILE_ATTRIBUTE_DIRECTORY))@@ -5623,8 +5668,6 @@ void Grep::search(const char *pathname) rest_line_size = rest_line.size(); rest_line_last = matcher->last(); }-- lineno += matcher->lines() - 1; } } else@@ -5690,9 +5733,9 @@ void Grep::search(const char *pathname) rest_line_size = rest_line.size(); rest_line_last = matcher->last(); }-- lineno += matcher->lines() - 1; }++ lineno += matcher->lines() - 1; } else {@@ -6775,6 +6818,8 @@ void help(const char *message, const char *arg) slow as every file on the search path is read to compare MAGIC.\n\ -m NUM, --max-count=NUM\n\ Stop reading the input after NUM matches for each file processed.\n\+ --match\n\+ Match input. Same as specifying an empty pattern to search.\n\ --max-depth=NUM\n\ Restrict recursive search to NUM (NUM > 0) directories deep, where\n\ --max-depth=1 searches the specified path without visiting\n\@@ -6815,7 +6860,7 @@ void help(const char *message, const char *arg) Interpret PATTERN as a Perl regular expression.\n"; #ifndef HAVE_BOOST_REGEX std::cout << "\- This feature is not available in this version of ugrep.\n";+ This feature is not available in this build configuration of ugrep.\n"; #endif std::cout << "\ -p, --no-dereference\n\@@ -6923,7 +6968,7 @@ void help(const char *message, const char *arg) searches files with matching file pathnames in archives.\n"; #ifndef HAVE_LIBZ std::cout << "\- This feature is not available in this version of ugrep.\n";+ This feature is not available in this build configuration of ugrep.\n"; #else std::cout << "\ Supports gzip (optional suffix .gz), compress (requires suffix .Z)";