![[BACK]](/icons/back.gif){kind=icon}
Up to [cvs.NetBSD.org] / pkgsrc / textproc / miller
Request diff between arbitrary revisions
Keyword substitution: kv
Default branch: MAIN
Revbump all Go packages after go122 update
Revbump all Go packages after update
Revbump all Go packages after go122 security update
Revbump all Go packages after go122 update
Revbump all Go packages, default Go version is now 1.22.
Revbump all Go packages after go121 update
Revbump all Go packages after go121 update
Revbump all Go packages after go121 update
Pullup ticket #6822 - requested by bsiegert
lang/go120: security update
lang/go121: security update
lang/go: metadata update
audio/gospt: revision bump
audio/ymuse: revision bump
chat/coyim: revision bump
chat/gomuks: revision bump
chat/matterircd: revision bump
chat/senpai: revision bump
chat/ssh-chat: revision bump
databases/go-ldap: revision bump
databases/influxdb: revision bump
databases/mongo-tools: revision bump
databases/mysqld_exporter: revision bump
databases/postgres_exporter: revision bump
databases/prometheus: revision bump
databases/promscale: revision bump
databases/sqlc: revision bump
databases/timescaledb-tune: revision bump
devel/asmfmt: revision bump
devel/conftest: revision bump
devel/errcheck: revision bump
devel/fq: revision bump
devel/git-lfs: revision bump
devel/go-ed25519: revision bump
devel/go-gocode: revision bump
devel/go-golang-lru: revision bump
devel/go-gopkgs: revision bump
devel/go-goptlib: revision bump
devel/go-goreturns: revision bump
devel/go-gox: revision bump
devel/go-impl: revision bump
devel/go-logrus: revision bump
devel/go-nbreader: revision bump
devel/go-pty: revision bump
devel/go-review: revision bump
devel/go-siphash: revision bump
devel/go-staticcheck: revision bump
devel/go-swagger: revision bump
devel/go-sys: revision bump
devel/go-tools: revision bump
devel/go-wire: revision bump
devel/go-xerrors: revision bump
devel/golangci-lint: revision bump
devel/golint: revision bump
devel/gomodifytags: revision bump
devel/gopls: revision bump
devel/goredo: revision bump
devel/gotags: revision bump
devel/gotests: revision bump
devel/govulncheck: revision bump
devel/lazygit: revision bump
devel/mob: revision bump
devel/nancy: revision bump
devel/opa: revision bump
devel/packr: revision bump
devel/reftools: revision bump
devel/regal: revision bump
devel/revive: revision bump
devel/shfmt: revision bump
devel/syft: revision bump
editors/micro: revision bump
filesystems/kubo: revision bump
graphics/gif2png: revision bump
lang/joker: revision bump
mail/opensmtpd-filter-rspamd: revision bump
mail/opensmtpd-filter-senderscore: revision bump
mail/postforward: revision bump
misc/exercism: revision bump
net/amazon-ecs-cli: revision bump
net/amfora: revision bump
net/bombadillo: revision bump
net/croc: revision bump
net/czds: revision bump
net/dnscontrol: revision bump
net/dnscrypt-proxy2: revision bump
net/gh: revision bump
net/go-dnstap: revision bump
net/go-net: revision bump
net/go-websocket: revision bump
net/gunison: revision bump
net/gvproxy: revision bump
net/hub: revision bump
net/ipget: revision bump
net/kubectl: revision bump
net/libquic: revision bump
net/mangos: revision bump
net/nats-server: revision bump
net/obfs4proxy: revision bump
net/rclone: revision bump
net/stern: revision bump
net/syncthing: revision bump
net/terraform-provider-archive: revision bump
net/terraform-provider-aws: revision bump
net/terraform-provider-kubernetes: revision bump
net/terraform-provider-local: revision bump
net/terraform-provider-null: revision bump
net/terraform-provider-random: revision bump
net/terraform-provider-template: revision bump
net/terraform-provider-vultr: revision bump
net/terraform: revision bump
net/tut: revision bump
net/vultr-cli: revision bump
pkgtools/pkglint: revision bump
security/2fa: revision bump
security/age: revision bump
security/amass: revision bump
security/authelia: revision bump
security/cfssl: revision bump
security/dnsx: revision bump
security/go-asn1-ber: revision bump
security/go-crypto: revision bump
security/go-getpass: revision bump
security/go-mkcert: revision bump
security/gopass: revision bump
security/httpx: revision bump
security/nuclei: revision bump
security/oauth2c: revision bump
security/osv-scanner: revision bump
security/subfinder: revision bump
security/tlsx: revision bump
security/trufflehog: revision bump
security/vault: revision bump
shells/elvish: revision bump
shells/oh-my-posh: revision bump
sysutils/beats: revision bump
sysutils/consul: revision bump
sysutils/direnv: revision bump
sysutils/fzf: revision bump
sysutils/goreman: revision bump
sysutils/lf: revision bump
sysutils/node_exporter: revision bump
sysutils/packer: revision bump
sysutils/podman: revision bump
sysutils/restic: revision bump
sysutils/vultr: revision bump
textproc/glow: revision bump
textproc/go-kr-text: revision bump
textproc/go-md2man: revision bump
textproc/go-mmark: revision bump
textproc/go-text: revision bump
textproc/miller: revision bump
textproc/sift: revision bump
www/apisprout: revision bump
www/caddy: revision bump
www/gitea: revision bump
www/go-ffuf: revision bump
www/go-minify: revision bump
www/gotosocial: revision bump
www/grafana: revision bump
www/hugo: revision bump
www/jira-cli: revision bump
www/mycorrhiza: revision bump
www/pup: revision bump
www/restish: revision bump
www/shoutrrr: revision bump
Revisions pulled up:
- lang/go/version.mk 1.194
- lang/go120/PLIST 1.10
- lang/go120/distinfo 1.12
- lang/go121/PLIST 1.4
- lang/go121/distinfo 1.4
- audio/gospt/Makefile by patch
- audio/ymuse/Makefile by patch
- chat/coyim/Makefile by patch
- chat/gomuks/Makefile by patch
- chat/matterircd/Makefile by patch
- chat/senpai/Makefile by patch
- chat/ssh-chat/Makefile by patch
- databases/go-ldap/Makefile by patch
- databases/influxdb/Makefile by patch
- databases/mongo-tools/Makefile by patch
- databases/mysqld_exporter/Makefile by patch
- databases/postgres_exporter/Makefile by patch
- databases/prometheus/Makefile by patch
- databases/promscale/Makefile by patch
- databases/sqlc/Makefile by patch
- databases/timescaledb-tune/Makefile by patch
- devel/asmfmt/Makefile by patch
- devel/conftest/Makefile by patch
- devel/errcheck/Makefile by patch
- devel/fq/Makefile by patch
- devel/git-lfs/Makefile by patch
- devel/go-ed25519/Makefile by patch
- devel/go-gocode/Makefile by patch
- devel/go-golang-lru/Makefile by patch
- devel/go-gopkgs/Makefile by patch
- devel/go-goptlib/Makefile by patch
- devel/go-goreturns/Makefile by patch
- devel/go-gox/Makefile by patch
- devel/go-impl/Makefile by patch
- devel/go-logrus/Makefile by patch
- devel/go-nbreader/Makefile by patch
- devel/go-pty/Makefile by patch
- devel/go-review/Makefile by patch
- devel/go-siphash/Makefile by patch
- devel/go-staticcheck/Makefile by patch
- devel/go-swagger/Makefile by patch
- devel/go-sys/Makefile by patch
- devel/go-tools/Makefile by patch
- devel/go-wire/Makefile by patch
- devel/go-xerrors/Makefile by patch
- devel/golangci-lint/Makefile by patch
- devel/golint/Makefile by patch
- devel/gomodifytags/Makefile by patch
- devel/gopls/Makefile by patch
- devel/goredo/Makefile by patch
- devel/gotags/Makefile by patch
- devel/gotests/Makefile by patch
- devel/govulncheck/Makefile by patch
- devel/lazygit/Makefile by patch
- devel/mob/Makefile by patch
- devel/nancy/Makefile by patch
- devel/opa/Makefile by patch
- devel/packr/Makefile by patch
- devel/reftools/Makefile by patch
- devel/regal/Makefile by patch
- devel/revive/Makefile by patch
- devel/shfmt/Makefile by patch
- devel/syft/Makefile by patch
- editors/micro/Makefile by patch
- filesystems/kubo/Makefile by patch
- graphics/gif2png/Makefile by patch
- lang/joker/Makefile by patch
- mail/opensmtpd-filter-rspamd/Makefile by patch
- mail/opensmtpd-filter-senderscore/Makefile by patch
- mail/postforward/Makefile by patch
- misc/exercism/Makefile by patch
- net/amazon-ecs-cli/Makefile by patch
- net/amfora/Makefile by patch
- net/bombadillo/Makefile by patch
- net/croc/Makefile by patch
- net/czds/Makefile by patch
- net/dnscontrol/Makefile by patch
- net/dnscrypt-proxy2/Makefile by patch
- net/gh/Makefile by patch
- net/go-dnstap/Makefile by patch
- net/go-net/Makefile by patch
- net/go-websocket/Makefile by patch
- net/gunison/Makefile by patch
- net/gvproxy/Makefile by patch
- net/hub/Makefile by patch
- net/ipget/Makefile by patch
- net/kubectl/Makefile by patch
- net/libquic/Makefile by patch
- net/mangos/Makefile by patch
- net/nats-server/Makefile by patch
- net/obfs4proxy/Makefile by patch
- net/rclone/Makefile by patch
- net/stern/Makefile by patch
- net/syncthing/Makefile by patch
- net/terraform-provider-archive/Makefile by patch
- net/terraform-provider-aws/Makefile by patch
- net/terraform-provider-kubernetes/Makefile by patch
- net/terraform-provider-local/Makefile by patch
- net/terraform-provider-null/Makefile by patch
- net/terraform-provider-random/Makefile by patch
- net/terraform-provider-template/Makefile by patch
- net/terraform-provider-vultr/Makefile by patch
- net/terraform/Makefile by patch
- net/tut/Makefile by patch
- net/vultr-cli/Makefile by patch
- pkgtools/pkglint/Makefile by patch
- security/2fa/Makefile by patch
- security/age/Makefile by patch
- security/amass/Makefile by patch
- security/authelia/Makefile by patch
- security/cfssl/Makefile by patch
- security/dnsx/Makefile by patch
- security/go-asn1-ber/Makefile by patch
- security/go-crypto/Makefile by patch
- security/go-getpass/Makefile by patch
- security/go-mkcert/Makefile by patch
- security/gopass/Makefile by patch
- security/httpx/Makefile by patch
- security/nuclei/Makefile by patch
- security/oauth2c/Makefile by patch
- security/osv-scanner/Makefile by patch
- security/subfinder/Makefile by patch
- security/tlsx/Makefile by patch
- security/trufflehog/Makefile by patch
- security/vault/Makefile by patch
- shells/elvish/Makefile by patch
- shells/oh-my-posh/Makefile by patch
- sysutils/beats/Makefile by patch
- sysutils/consul/Makefile by patch
- sysutils/direnv/Makefile by patch
- sysutils/fzf/Makefile by patch
- sysutils/goreman/Makefile by patch
- sysutils/lf/Makefile by patch
- sysutils/node_exporter/Makefile by patch
- sysutils/packer/Makefile by patch
- sysutils/podman/Makefile by patch
- sysutils/restic/Makefile by patch
- sysutils/vultr/Makefile by patch
- textproc/glow/Makefile by patch
- textproc/go-kr-text/Makefile by patch
- textproc/go-md2man/Makefile by patch
- textproc/go-mmark/Makefile by patch
- textproc/go-text/Makefile by patch
- textproc/miller/Makefile by patch
- textproc/sift/Makefile by patch
- www/apisprout/Makefile by patch
- www/caddy/Makefile by patch
- www/gitea/Makefile by patch
- www/go-ffuf/Makefile by patch
- www/go-minify/Makefile by patch
- www/gotosocial/Makefile by patch
- www/grafana/Makefile by patch
- www/hugo/Makefile by patch
- www/jira-cli/Makefile by patch
- www/mycorrhiza/Makefile by patch
- www/pup/Makefile by patch
- www/restish/Makefile by patch
- www/shoutrrr/Makefile by patch
-------------------------------------------------------------------
Module Name: pkgsrc
Committed By: bsiegert
Date: Fri Nov 10 15:39:34 UTC 2023
Modified Files:
pkgsrc/lang/go: version.mk
pkgsrc/lang/go120: PLIST distinfo
pkgsrc/lang/go121: PLIST distinfo
Log Message:
Update go120 to 1.20.11 and go121 to 1.21.4 (security).
These minor releases include 2 security fixes following the security policy:
- path/filepath: recognize \??\ as a Root Local Device path prefix.
On Windows, a path beginning with \??\ is a Root Local Device path equivalent
to a path beginning with \\?\. Paths with a \??\ prefix may be used to access
arbitrary locations on the system. For example, the path \??\c:\x is
equivalent to the more common path c:\x.
The filepath package did not recognize paths with a \??\ prefix as special.
Clean could convert a rooted path such as \a\..\??\b into
the root local device path \??\b. It will now convert this
path into .\??\b.
IsAbs did not report paths beginning with \??\ as absolute.
It now does so.
VolumeName now reports the \??\ prefix as a volume name.
Join(`\`, `??`, `b`) could convert a seemingly innocent
sequence of path elements into the root local device path
\??\b. It will now convert this to \.\??\b.
This is CVE-2023-45283 and https://go.dev/issue/63713.
- path/filepath: recognize device names with trailing spaces and superscripts
The IsLocal function did not correctly detect reserved names in some cases:
reserved names followed by spaces, such as "COM1 ".
"COM" or "LPT" followed by a superscript 1, 2, or 3.
IsLocal now correctly reports these names as non-local.
This is CVE-2023-45284 and https://go.dev/issue/63713.
To generate a diff of this commit:
cvs rdiff -u -r1.193 -r1.194 pkgsrc/lang/go/version.mk
cvs rdiff -u -r1.9 -r1.10 pkgsrc/lang/go120/PLIST
cvs rdiff -u -r1.11 -r1.12 pkgsrc/lang/go120/distinfo
cvs rdiff -u -r1.3 -r1.4 pkgsrc/lang/go121/PLIST pkgsrc/lang/go121/distinfo
Revbump all Go packages after go121 update
Revbump all Go packages after go121 update
Revbump all Go packages because go121 is now the default
Pullup ticket #6812 - requested by bsiegert
lang/go120: security update
lang/go: version info update
audio/gospt: RevBump
audio/ymuse: RevBump
chat/coyim: RevBump
chat/gomuks: RevBump
chat/matterircd: RevBump
chat/senpai: RevBump
chat/ssh-chat: RevBump
databases/go-ldap: RevBump
databases/influxdb: RevBump
databases/mongo-tools: RevBump
databases/mysqld_exporter: RevBump
databases/postgres_exporter: RevBump
databases/prometheus: RevBump
databases/promscale: RevBump
databases/sqlc: RevBump
databases/timescaledb-tune: RevBump
devel/asmfmt: RevBump
devel/conftest: RevBump
devel/errcheck: RevBump
devel/fq: RevBump
devel/git-lfs: RevBump
devel/go-ed25519: RevBump
devel/go-gocode: RevBump
devel/go-golang-lru: RevBump
devel/go-gopkgs: RevBump
devel/go-goptlib: RevBump
devel/go-goreturns: RevBump
devel/go-gox: RevBump
devel/go-impl: RevBump
devel/go-logrus: RevBump
devel/go-nbreader: RevBump
devel/go-pty: RevBump
devel/go-review: RevBump
devel/go-siphash: RevBump
devel/go-staticcheck: RevBump
devel/go-swagger: RevBump
devel/go-sys: RevBump
devel/go-tools: RevBump
devel/go-wire: RevBump
devel/go-xerrors: RevBump
devel/golangci-lint: RevBump
devel/golint: RevBump
devel/gomodifytags: RevBump
devel/gopls: RevBump
devel/goredo: RevBump
devel/gotags: RevBump
devel/gotests: RevBump
devel/govulncheck: RevBump
devel/lazygit: RevBump
devel/mob: RevBump
devel/nancy: RevBump
devel/opa: RevBump
devel/packr: RevBump
devel/reftools: RevBump
devel/regal: RevBump
devel/revive: RevBump
devel/shfmt: RevBump
devel/syft: RevBump
editors/micro: RevBump
filesystems/kubo: RevBump
graphics/gif2png: RevBump
lang/joker: RevBump
mail/opensmtpd-filter-rspamd: RevBump
mail/opensmtpd-filter-senderscore: RevBump
mail/postforward: RevBump
meta-pkgs/bulk-test-essential: RevBump
misc/exercism: RevBump
net/amazon-ecs-cli: RevBump
net/amfora: RevBump
net/bombadillo: RevBump
net/croc: RevBump
net/czds: RevBump
net/dnscontrol: RevBump
net/dnscrypt-proxy2: RevBump
net/gh: RevBump
net/go-dnstap: RevBump
net/go-net: RevBump
net/go-websocket: RevBump
net/gunison: RevBump
net/gvproxy: RevBump
net/hub: RevBump
net/ipget: RevBump
net/kubectl: RevBump
net/libquic: RevBump
net/mangos: RevBump
net/nats-server: RevBump
net/obfs4proxy: RevBump
net/rclone: RevBump
net/stern: RevBump
net/syncthing: RevBump
net/terraform-provider-archive: RevBump
net/terraform-provider-aws: RevBump
net/terraform-provider-kubernetes: RevBump
net/terraform-provider-local: RevBump
net/terraform-provider-null: RevBump
net/terraform-provider-random: RevBump
net/terraform-provider-template: RevBump
net/terraform-provider-vultr: RevBump
net/terraform: RevBump
net/tut: RevBump
net/vultr-cli: RevBump
pkgtools/pkglint: RevBump
security/2fa: RevBump
security/age: RevBump
security/amass: RevBump
security/authelia: RevBump
security/cfssl: RevBump
security/dnsx: RevBump
security/go-asn1-ber: RevBump
security/go-crypto: RevBump
security/go-getpass: RevBump
security/go-mkcert: RevBump
security/gopass: RevBump
security/httpx: RevBump
security/nuclei: RevBump
security/oauth2c: RevBump
security/osv-scanner: RevBump
security/subfinder: RevBump
security/tlsx: RevBump
security/trufflehog: RevBump
security/vault: RevBump
shells/elvish: RevBump
shells/oh-my-posh: RevBump
sysutils/beats: RevBump
sysutils/consul: RevBump
sysutils/direnv: RevBump
sysutils/fzf: RevBump
sysutils/goreman: RevBump
sysutils/lf: RevBump
sysutils/node_exporter: RevBump
sysutils/packer: RevBump
sysutils/podman: RevBump
sysutils/restic: RevBump
sysutils/vultr: RevBump
textproc/glow: RevBump
textproc/go-kr-text: RevBump
textproc/go-md2man: RevBump
textproc/go-mmark: RevBump
textproc/go-text: RevBump
textproc/miller: RevBump
textproc/sift: RevBump
www/apisprout: RevBump
www/caddy: RevBump
www/gitea: RevBump
www/go-ffuf: RevBump
www/go-minify: RevBump
www/gotosocial: RevBump
www/grafana: RevBump
www/hugo: RevBump
www/jira-cli: RevBump
www/mycorrhiza: RevBump
www/pup: RevBump
www/restish: RevBump
www/shoutrrr: RevBump
Revisions pulled up:
- lang/go/version.mk 1.190,1.192
- lang/go120/PLIST 1.9
- lang/go120/distinfo 1.10-1.11
-------------------------------------------------------------------
Module Name: pkgsrc
Committed By: bsiegert
Date: Sat Oct 7 18:09:35 UTC 2023
Modified Files:
pkgsrc/lang/go: version.mk
pkgsrc/lang/go120: PLIST distinfo
Log Message:
go120: update to 1.20.9 (security).
cmd/go: line directives allows arbitrary execution during build
"//line" directives can be used to bypass the restrictions on "//go:cgo_"
directives, allowing blocked linker and compiler flags to be passed during
compliation. This can result in unexpected execution of arbitrary code when
running "go build". The line directive requires the absolute path of the file in
which the directive lives, which makes exploting this issue significantly more
complex.
This is CVE-2023-39323 and Go issue https://go.dev/issue/63211.
View the release notes for more information:
https://go.dev/doc/devel/release#go1.20.9
To generate a diff of this commit:
cvs rdiff -u -r1.189 -r1.190 pkgsrc/lang/go/version.mk
cvs rdiff -u -r1.8 -r1.9 pkgsrc/lang/go120/PLIST
cvs rdiff -u -r1.9 -r1.10 pkgsrc/lang/go120/distinfo
-------------------------------------------------------------------
Module Name: pkgsrc
Committed By: bsiegert
Date: Sun Oct 15 11:02:08 UTC 2023
Modified Files:
pkgsrc/lang/go: version.mk
pkgsrc/lang/go120: distinfo
Log Message:
go120: update to 1.20.10 (security)
net/http: rapid stream resets can cause excessive work
A malicious HTTP/2 client which rapidly creates requests and
immediately resets them can cause excessive server resource consumption.
While the total number of requests is bounded to the
http2.Server.MaxConcurrentStreams setting, resetting an in-progress
request allows the attacker to create a new request while the existing
one is still executing.
HTTP/2 servers now bound the number of simultaneously executing
handler goroutines to the stream concurrency limit. New requests
arriving when at the limit (which can only happen after the client
has reset an existing, in-flight request) will be queued until a
handler exits. If the request queue grows too large, the server
will terminate the connection.
This issue is also fixed in golang.org/x/net/http2 v0.17.0,
for users manually configuring HTTP/2.
The default stream concurrency limit is 250 streams (requests)
per HTTP/2 connection. This value may be adjusted using the
golang.org/x/net/http2 package; see the Server.MaxConcurrentStreams
setting and the ConfigureServer function.
This is CVE-2023-39325 and Go issue https://go.dev/issue/63417.
This is also tracked by CVE-2023-44487.
To generate a diff of this commit:
cvs rdiff -u -r1.191 -r1.192 pkgsrc/lang/go/version.mk
cvs rdiff -u -r1.10 -r1.11 pkgsrc/lang/go120/distinfo
Revbump all Go packages after go120 security update
textproc/miller: update to miller-6.9.0
Changes:
6.9.0
-----
Nanosecond timestamps, ZSTD compression, improved data-error handling,
and more
New features
Support for nanosecond-resolution timestamps:
* Add DSL functions for integer nanoseconds since the epoch by
@johnkerl in #1326
* Add %N and %O for strfntime by @johnkerl in #1334
* Add %s format specifier for strftime by @johnkerl in #1335
* Requested on issue #1152 by @derekmahar
* See also https://miller.readthedocs.io/en/6.9.0/reference-dsl-builtin-functions/index.html#time-functions
Statistical functions from the stats verb may now be computed over
arbitrary arrays and maps:
* New DSL functions for summary stats over arrays / maps by
@johnkerl in #1364
* Requested on issue #1345 by @janxkoci
* See also https://miller.readthedocs.io/en/6.9.0/reference-dsl-builtin-functions/index.html#stats-functions
Additional control over filenames for the split verb:
* Filename options for split by @sloanlance in #1366
* Requested on issue #1365 by @sloanlance
Support for details of data-computation errors beyond the current
(error):
* Fatal-on-data-error mlr -x option by @johnkerl in #1373
* See also https://miller.readthedocs.io/en/6.9.0/reference-dsl-errors/#handling-for-data-errors
* Requested on issue #1106 by @honzajde
New verbs and DSL functions:
* New sub, gsub, and ssub verbs by @johnkerl in #1361, requested by
@janxkoci on #1356. See also:
https://miller.readthedocs.io/en/6.9.0/reference-verbs/#sub
https://miller.readthedocs.io/en/6.9.0/reference-verbs/#gsub
https://miller.readthedocs.io/en/6.9.0/reference-verbs/#ssub
* New contains DSL function by @johnkerl in #1374
https://miller.readthedocs.io/en/6.9.0/reference-dsl-builtin-functions/index.html#contains
Other updates:
* Support ZSTD compression in-process by @johnkerl in #1360, from
issue #1342 by @aborruso
See also https://miller.readthedocs.io/en/6.9.0/reference-main-compressed-data/
* Support comments in mlr -s files by @johnkerl in #1359, from
issue #1343 by @janxkoci
* Add empty-key check to mlr check by @johnkerl in #1330, from
issue #1050 by @aborruso
Bug fixes
* Do wildcard globbing on Windows by @johnkerl in #1362, from
issue #1341 by @coolbq
* Treat empty like absent in + - * by @johnkerl in #1371, from
issue #1001 by @Poshi
* Can't use ${field_name} if it contains UTF-8 characters also
encodeable as Latin-1 by @johnkerl in #1363, from issue #1358
by @clemente
* Typofix in uif/uof percentiles by @johnkerl in #1375
Documentation updates
* Update readthedocs notes in the how-to-release page by @johnkerl
in #1308
* Fix mlr grep docs re OFS/OPS by @johnkerl in #1309
* Update Fedora link by @bkmgit in #1339
* Small typos in documentation of mlr nest by @johnkerl in #1352,
from issue #1350 by @kusalananda
Internal
* Update 2015-era Python sketch to Python 3 by @johnkerl in #1372
* Remove redundant nil check by @Juneezee in #1367
* Bump actions/checkout from 3.5.2 to 3.5.3 by @dependabot in #1319
* Bump github/codeql-action from 2.3.6 to 2.13.4 by @dependabot in #1318
* Bump golang.org/x/term from 0.8.0 to 0.9.0 by @dependabot in #1321
* Bump goreleaser/goreleaser-action from 4.2.0 to 4.3.0 by @dependabot in #1320
* Bump golang.org/x/text from 0.9.0 to 0.10.0 by @dependabot in #1322
* Bump golang.org/x/text from 0.10.0 to 0.11.0 by @dependabot in #1337
* Bump golang.org/x/sys from 0.9.0 to 0.10.0 by @dependabot in #1336
* Bump golang.org/x/term from 0.9.0 to 0.10.0 by @dependabot in #1338
* Bump golang.org/x/sys from 0.10.0 to 0.11.0 by @dependabot in #1347
* Bump golang.org/x/text from 0.11.0 to 0.12.0 by @dependabot in #1349
* Bump actions/setup-go from 4.0.1 to 4.1.0 by @dependabot in #1351
* Bump goreleaser/goreleaser-action from 4.3.0 to 4.4.0 by @dependabot in #1354
* Bump golang.org/x/term from 0.10.0 to 0.11.0 by @dependabot in #1348
* Bump actions/checkout from 3.5.3 to 3.6.0 by @dependabot in #1369
New Contributors
* @bkmgit made their first contribution in #1339
* @Juneezee made their first contribution in #1367
* @sloanlance made their first contribution in #1366
6.8.0
-----
New case verb, index DSL function, and more
New features
New case verb:
* Unify the case verb, and add options by @johnkerl in #1306
* Add new upcase and downcase verbs by @johnkerl in #1217
New index DSL function:
* index DSL function by @johnkerl in #1247
Enhancements:
* Add mlr step -a rprod for running products by @johnkerl in #1228
* Add optional second base argument to int DSL function
by @johnkerl in #1244
* Implement --csv-trim-leading-space flag by @johnkerl in #1272
* New mlr json-parse -k flag by @johnkerl in #1291
* Let mlr help take pre-flags, such as --always-color
by @johnkerl in #1292
* Values-only -a option for mlr grep by @johnkerl in #1305
Bugfixes
* Fix bug on DSL comment with no final newline by @johnkerl in #1216
* Better error message on unparseable TZ environment variable
by @johnkerl in #1249
* Fix typo by @dnicolson in #1252
* Treat data-file numbers with leading + as numeric
by @johnkerl in #1269
* Fix precedence of coalesce operators ?? and ??? by @johnkerl
in #1270
Documentation
* Docs re tail -f and --records-per-batch 1 by @johnkerl in #1218
* Fix issue links in README-dev.md by @kcwu in #1248
Miscellaneous
* Include tools in the release tarball by @skitt in #1221
* Run go mod tidy by @skitt in #1220
Dependencies
Go:
* Bump minimum compiler version from Go 1.15 to 1.18 by @johnkerl in #1246
Others:
* Bump github/codeql-action from 2.2.5 to 2.2.6 by @dependabot in #1230
* Bump actions/cache from 3.2.6 to 3.3.1 by @dependabot in #1229
* Bump github/codeql-action from 2.2.6 to 2.2.7 by @dependabot in #1232
* Bump actions/setup-go from 3.5.0 to 4.0.0 by @dependabot in #1233
* Bump actions/checkout from 3.3.0 to 3.4.0 by @dependabot in #1234
* Bump github/codeql-action from 2.2.7 to 2.2.8 by @dependabot in #1242
* Bump actions/checkout from 3.4.0 to 3.5.0 by @dependabot in #1245
* Bump golang.org/x/term from 0.0.0-20210927222741-03fcf44c2211 to 0.6.0 by @dependabot in #1222
* Bump github.com/mattn/go-isatty from 0.0.17 to 0.0.18 by @dependabot in #1243
* Bump github/codeql-action from 2.2.8 to 2.2.9 by @dependabot in #1250
* Bump codespell-project/actions-codespell from 9c63fddd79f483308bfaea379a505dcd361b5d1d to 57beb9f38f49d773d641ac555d1565c3b6a59938 by @dependabot in #1253
* Bump golang.org/x/term from 0.6.0 to 0.7.0 by @dependabot in #1256
* Bump github/codeql-action from 2.2.9 to 2.2.10 by @dependabot in #1259
* Bump github/codeql-action from 2.2.10 to 2.2.11 by @dependabot in #1261
* Bump actions/checkout from 3.5.0 to 3.5.1 by @dependabot in #1263
* Bump actions/checkout from 3.5.1 to 3.5.2 by @dependabot in #1264
* Bump github/codeql-action from 2.2.11 to 2.2.12 by @dependabot in #1265
* Bump github/codeql-action from 2.2.12 to 2.3.0 by @dependabot in #1274
* Bump github/codeql-action from 2.3.0 to 2.3.1 by @dependabot in #1277
* Bump github/codeql-action from 2.3.1 to 2.3.2 by @dependabot in #1279
* Bump codespell-project/actions-codespell from 57beb9f38f49d773d641ac555d1565c3b6a59938 to 94259cd8be02ad2903ba34a22d9c13de21a74461 by @dependabot in #1282
* Bump github/codeql-action from 2.3.2 to 2.3.3 by @dependabot in #1284
* Bump golang.org/x/term from 0.7.0 to 0.8.0 by @dependabot in #1285
* Bump actions/setup-go from 4.0.0 to 4.0.1 by @dependabot in #1294
* Bump github.com/stretchr/testify from 1.8.2 to 1.8.3 by @dependabot in #1295
* Bump github.com/mattn/go-isatty from 0.0.18 to 0.0.19 by @dependabot in #1296
* Bump github/codeql-action from 2.3.3 to 2.3.4 by @dependabot in #1299
* Bump github/codeql-action from 2.3.4 to 2.3.5 by @dependabot in #1300
* Bump github.com/stretchr/testify from 1.8.3 to 1.8.4 by @dependabot in #1301
* Bump github/codeql-action from 2.3.5 to 2.3.6 by @dependabot in #1303
New Contributors
* @kcwu made their first contribution in #1248
* @dnicolson made their first contribution in #1252
Revbump all Go packages after go120 update
Revbump all Go packages after go120 update
Revbump all Go packages after go120 update
textproc/miller: update to miller-6.7.0
v6.7.0
======
New leftpad/rightpad DSL functions, unspace verb, and more
Features
* New leftpad and rightpad DSL functions by @johnkerl in #1205
* mlr unspace verb by @johnkerl in #1167
* Support more backslashed special characters in DSL strings by @johnkerl in #1212
* Add --ofmte, --ofmtf, --ofmtg command-line flags by @johnkerl in #1206
Documentation updates
* Fixed missing double quote in documentation sample by @Clindbergh in #1181
* Complete #1181 by @johnkerl in #1184
* Add doc info on DSL code-comment syntax by @johnkerl in #1165
* Fix typos by @jwilk in #1135
* Fix typo in mlr put documentation by @johnkerl in #1140
Bug fixes
* Fix #1164: regression on CSV blank-line handling by @johnkerl in #1168
* Fix #1146: bug in lo/hi limits for non-auto histogram by @johnkerl in #1157
* Fix #1102: empty-string field in single-column TSV should not be a schema-restart by @johnkerl in #1163
Minor changes
* Add Go LICENSE file by @skitt in #1171
Dependency updates
* Bump github/codeql-action from 2.1.33 to 2.1.35 by @dependabot in #1137
* Bump actions/setup-go from 3.3.1 to 3.4.0 by @dependabot in #1136
* Bump github/codeql-action from 2.1.35 to 2.1.36 by @dependabot in #1143
* Bump actions/checkout from 3.1.0 to 3.2.0 by @dependabot in #1145
* Bump goreleaser/goreleaser-action from 3.2.0 to 4.1.0 by @dependabot in #1147
* Bump actions/setup-go from 3.4.0 to 3.5.0 by @dependabot in #1148
* Bump github/codeql-action from 2.1.36 to 2.1.37 by @dependabot in #1151
* Bump actions/cache from 3.0.11 to 3.2.0 by @dependabot in #1155
* Bump actions/cache from 3.2.0 to 3.2.1 by @dependabot in #1156
* Bump actions/cache from 3.2.1 to 3.2.2 by @dependabot in #1160
* Bump github.com/mattn/go-isatty from 0.0.16 to 0.0.17 by @dependabot in #1162
* Bump codespell-project/actions-codespell from bcf481f4d5cce7b92b65f05aebe8f552d4f1442c to
9c63fddd79f483308bfaea379a505dcd361b5d1d by @dependabot in #1172
* Bump actions/checkout from 3.2.0 to 3.3.0 by @dependabot in #1173
* Bump actions/cache from 3.2.2 to 3.2.3 by @dependabot in #1174
* Bump actions/upload-artifact from 3.1.1 to 3.1.2 by @dependabot in #1175
* Bump github/codeql-action from 2.1.37 to 2.1.38 by @dependabot in #1176
* Bump github/codeql-action from 2.1.38 to 2.1.39 by @dependabot in #1179
* Bump github/codeql-action from 2.1.39 to 2.2.1 by @dependabot in #1183
* Bump goreleaser/goreleaser-action from 4.1.0 to 4.1.1 by @dependabot in #1185
* Bump goreleaser/goreleaser-action from 4.1.1 to 4.2.0 by @dependabot in #1187
* Bump actions/cache from 3.2.3 to 3.2.4 by @dependabot in #1186
* Bump actions/cache from 3.2.4 to 3.2.5 by @dependabot in #1192
* Bump github/codeql-action from 2.2.1 to 2.2.3 by @dependabot in #1191
* Bump github/codeql-action from 2.2.3 to 2.2.4 by @dependabot in #1193
* Bump actions/cache from 3.2.5 to 3.2.6 by @dependabot in #1196
* Bump golang.org/x/sys from 0.0.0-20210326220804-49726bf1d181 to 0.1.0 in /cmd/experiments/cli_parser by @dependabot in #1203
* Bump github.com/stretchr/testify from 1.8.1 to 1.8.2 by @dependabot in #1208
* Bump github/codeql-action from 2.2.4 to 2.2.5 by @dependabot in #1207
New Contributors
* @jwilk made their first contribution in #1135
* @Clindbergh made their first contribution in #1181
v6.6.0
======
What's Changed
Features
* mlr unspace verb by @johnkerl in #1167
Bugfixes
* Add doc info on DSL code-comment syntax by @johnkerl in #1165
* Fix typos by @jwilk in #1135
* Fix typo in mlr put documentation by @johnkerl in #1140
* Fix #1146: bug in lo/hi limits for non-auto histogram by @johnkerl in #1157
* Fix #1102: empty-string field in single-column TSV should not be a schema-restart by @johnkerl in #1163
* Fix #1164: regression on CSV blank-line handling by @johnkerl in #1168
Internal
* Bump github/codeql-action from 2.1.33 to 2.1.35 by @dependabot in #1137
* Bump actions/setup-go from 3.3.1 to 3.4.0 by @dependabot in #1136
* Bump github/codeql-action from 2.1.35 to 2.1.36 by @dependabot in #1143
* Bump actions/checkout from 3.1.0 to 3.2.0 by @dependabot in #1145
* Bump goreleaser/goreleaser-action from 3.2.0 to 4.1.0 by @dependabot in #1147
* Bump actions/setup-go from 3.4.0 to 3.5.0 by @dependabot in #1148
* Bump github/codeql-action from 2.1.36 to 2.1.37 by @dependabot in #1151
* Bump actions/cache from 3.0.11 to 3.2.0 by @dependabot in #1155
* Bump actions/cache from 3.2.0 to 3.2.1 by @dependabot in #1156
* Bump actions/cache from 3.2.1 to 3.2.2 by @dependabot in #1160
* Bump github.com/mattn/go-isatty from 0.0.16 to 0.0.17 by @dependabot in #1162
v6.5.0
======
What's Changed
Features:
* Restore the --jvquoteall flag by @johnkerl in #1083
* Restore --quote-all for CSV output by @johnkerl in #1084
Bugfixes:
* Fix labels for mlr histogram --auto by @johnkerl in #1089
* Correctly support multiple regexes in mlr reshape -r by @johnkerl in #1091
* Check -- terminator on --mfrom by @johnkerl in #1098
* Type-safety in exec by @johnkerl in #1099
* Don't double-quote a CSV field only for having a leading space by @johnkerl in #1101
Performance/memory-reduction:
* Use int8 for mvtype (memory reduction) by @johnkerl in #1130
* Exclude median from summary default by @johnkerl in #1131
* More mlrval size-reduction by @johnkerl in #1132
* Convert mlrval polymorphism from struct to unionish interface by @johnkerl in #1133
Minor/internal:
* Account for varying mlr locations by @skitt in #1086
* Account for varying mlr locations, continued by @johnkerl in #1087
* [StepSecurity] ci: Harden GitHub Actions by @step-security-bot in #1107
* Bump github.com/pkg/profile from 1.6.0 to 1.7.0 by @dependabot in #1110
* Bump github/codeql-action from 2.1.28 to 2.1.33 by @dependabot in #1126
* Bump actions/cache from 3 to 3.0.11 by @dependabot in #1109
* Bump actions/upload-artifact from 3.1.0 to 3.1.1 by @dependabot in #1112
* Bump github.com/stretchr/testify from 1.8.0 to 1.8.1 by @dependabot in #1113
* Miller 6.5.0 by @johnkerl in #1134
v5.10.4
=======
Miller 5 is long-gone; 6.0.0 was released almost a year ago. Yet issue #1108 reports a critical memory-corruption bug on 5.10.3;
this fixes that.
v6.4.0
======
What's Changed
Major:
* mlr summary verb by @johnkerl in #1056
* feat: system/exec() function call ( #1043) by @forbesmyester in #1067 and #1071
* Support simplified sort-map-by-value in the DSL by @johnkerl in #1069
* mlr cat --filename / --filenum by @johnkerl in #1080
* Allow multi-line string literals in the DSL by @johnkerl in #1070
Minor:
* Make PPRINT empty-string markers readable as such by @johnkerl in #1059
* Allow "\n" in mlr repl prompt by @johnkerl in #1058
Bugfixes:
* [Docs] moving --xvright out of the FLATTEN-UNFLATTEN FLAGS section by @trantor in #1065
* Fix doc typo by @luzpaz in #1054
* Fix natsort of empty strings; support mlr sort -rt same as -tr by @johnkerl in #1068
Internal:
* Reduce number of os.Exit callsites, part 1 of n by @johnkerl in #1055
* delete unreachable test code caused by os.Exit by @Abirdcfly in #1073
* Bump github.com/mattn/go-isatty from 0.0.14 to 0.0.16 by @dependabot in #1074
v6.3.0
======
What's Changed
Key feature: output colorization on Windows thanks to @tiesmaster:
* Enable ANSI escape-sequence processing on Windows by @tiesmaster in #1045
* Enable output colorization on Windows by default by @johnkerl in #1051
Support for Latin-1:
* DSL functions and verbs for UTF-8 <-> Latin-1 by @johnkerl in #997
Features:
* Re-use whitespace regexp in clean_whitespace by @johnkerl in #994
* Add line/column info for DSL runtime non-parse failures by @johnkerl in #998
* Allow x ** - y and x ** + y in the DSL grammar by @johnkerl in #1021
* Let + be an alias for then by @johnkerl in #1049
Docs:
* helm/kubectl examples in webdocs by @johnkerl in #1005
Bugfixes:
* Accept + in exponent of scientific-notation floating-point DSL literals by @johnkerl in #1020
* Fix ASCII vs UTF-8 in TSV writer by @johnkerl in #1023
* Avoid panic when the command line ends in 'then' by @johnkerl in #1033
* Fix panic on 'mlr sort -n' by @johnkerl in #1004
* Fix issue 1037 by @johnkerl in #1047
* Fix issue 1032 by @johnkerl in #1048
Dependencies:
* Bump actions/cache from 2 to 3 by @dependabot in #1000
* Bump github.com/stretchr/testify from 1.7.1 to 1.7.2 by @dependabot in #1034
* Bump github.com/stretchr/testify from 1.7.2 to 1.7.3 by @dependabot in #1038
* Bump github.com/stretchr/testify from 1.7.3 to 1.7.4 by @dependabot in #1040
* Bump github.com/stretchr/testify from 1.7.4 to 1.7.5 by @dependabot in #1042
* Bump github.com/stretchr/testify from 1.7.5 to 1.8.0 by @dependabot in #1044
* Bump actions/upload-artifact from 2 to 3 by @dependabot in #1010
* Bump actions/setup-go from 2 to 3 by @dependabot in #1009
* Bump github.com/lestrrat-go/strftime from 1.0.5 to 1.0.6 by @dependabot in #1012
* Bump github/codeql-action from 1 to 2 by @dependabot in #1015
* Bump goreleaser/goreleaser-action from 2 to 3 by @dependabot in #1027
v6.2.0
======
Overview
The primary purpose of this release is to restore --tsvlite which, by its own, would merit a 6.1.1 bugfix release. But since a
couple other new features are present as well, this is a 6.2.0 minor release.
All the "Plans for 6.2.0" listed at https://github.com/johnkerl/miller/releases/tag/v6.1.0 are all still in-plan, but since this
6.2.0 exists sooner than later, those issues are planned for a 6.3.0.
Details
PRs:
* Restore --tsvlite by @johnkerl in #984
* Let dhms2sec accept input like "8h" by @johnkerl in #983
* Use fixed OFMT for multi-platform regression-testing by @johnkerl in #988
* Bump github.com/stretchr/testify from 1.7.0 to 1.7.1 by @dependabot in #986
* gssub DSL function by @johnkerl in #989
v6.1.0
======
Features
Major features:
* Natural sort by @johnkerl in #932
* mlr split verb by @johnkerl in #898
* Make TSV finally true TSV by @johnkerl in #923
* Sliding window averages by @johnkerl in #894
* Implement shift-lead option for mlr step by @johnkerl in #893
New DSL functions:
* New fmtifnum DSL function; make fmtnum/fmtifnum recursive over maps and arrays by @johnkerl in #946
* New unformat DSL function by @johnkerl in #871
* New format DSL function by @johnkerl in #869
* New concat DSL function for arrays by @johnkerl in #868
DSL improvements:
* Support more Go regex patterns, like "\d" by @johnkerl in #974
* Include \U support in addition to \u for DSL Unicode string literals by @johnkerl in #917
* Support unicode literals in the Miller DSL by @johnkerl in #916
* Allow 0o... octal literals in the DSL by @johnkerl in #864
New command-line flags:
* Add --left-keep-fields option for mlr join by @johnkerl in #967
* New --lazy-quotes flag for helping with malformed CSV by @johnkerl in #925
REPL and on-line help:
* Let :resetblocks/:rb in the REPL take optional begin/main/end by @johnkerl in #924
* Add :resetblocks / :rb to REPL by @johnkerl in #920
* ?foo and ??foo for :help foo / :help find foo in the REPL by @johnkerl in #915
Improvements and bugfixes
* Support Latin-1 supplement a0-ff as DSL string literals by @johnkerl in #957
* Fix "%%" in strptime; more test cases for strptime by @johnkerl in #951
* Support %F, %T, and more in strptime by @johnkerl in #944
* Fix handling of mlr nest abbrevs by @johnkerl in #937
* Add Inf and NaN literals to the DSL by @johnkerl in #933
* Boolean inference for issue 908 by @johnkerl in #931
* strptime %j format for 3-digit day in year by @johnkerl in #930
* Fix is_non_empty for absent case by @johnkerl in #928
* --nidx --fs x should be the same as --fs x --nidx by @johnkerl in #912
* Update default colorization by @johnkerl in #904
* Make is_null/is_not_null DSL functions include new JSON-null type by @johnkerl in #883
* Fix #853 by @johnkerl in #860
Documentation
* New doc page: Parsing and formatting fields by @johnkerl in #973
* More doc material for :context in the REPL by @johnkerl in #966
* Fix typo in on-line help for splitax DSL function by @johnkerl in #964
* More doc-sites for the funct keyword by @johnkerl in #963
* Doc updates for funct keyword by @johnkerl in #961
* FAQ entry for #351 by @johnkerl in #958
* docs: add Poshi as a contributor for doc by @allcontributors in #956
* docs: add schragge as a contributor for doc by @allcontributors in #955
* FAQ entry for #285: carriage returns in field names by @johnkerl in #953
* Add --implicit-tsv-header as alias for --implicit-csv-header, etc by @johnkerl in #952
* Fix: multiple documentation tweaks by @Poshi in #949
* fix typo in reference-verbs.md by @zachvalenta in #945
* Add on mouse over permalink anchor for titles by @aborruso in #942
* Webdoc information on Unicode string literals by @johnkerl in #935
* 'mlr help function nonesuch' should not be silent by @johnkerl in #934
* Clarify strftime on-line help by @johnkerl in #929
* Expand on-line help for split* DSL functions by @johnkerl in #927
* On-line help for -s flag by @johnkerl in #926
* Multiple on-line-help issues from #908 by @johnkerl in #921
* Multiple on-line-help issues from #908 by @johnkerl in #913
* Fix operator-precedence doc table to match DSL grammar by @johnkerl in #911
* Fix multiple on-line-help issues from #907 by @johnkerl in #910
* Clarify source for printf-style formatting by @johnkerl in #895
* Fix #891 by @johnkerl in #892
* Improve mlr top documentation for #861 by @johnkerl in #875
* Continue #856 by @johnkerl in #865
* misspelling by @Gary-Armstrong in #863
* fix typo by @vapniks in #862
* Update installing-miller.md by @jauderho in #859
* Emit notes by @johnkerl in #858
* Conda/Docker install notes by @johnkerl in #857
* Fix typo: columnn -> column by @vapniks in #856
* Fix typo by @vapniks in #855
* Fix typo by @vapniks in #854
* A small typo by @aborruso in #846
Code quality
* Code-dedupe logic for array slices and string slices by @johnkerl in #960
* Let mlr repl print empty strings by @johnkerl in #959
* Neaten strptime.go by @johnkerl in #950
* More dead code removal by @skitt in #905
* Remove unreachable code by @skitt in #903
* Use int64 wherever "64-bit integer" is assumed by @skitt in #902
* More of #884: types in enum-consts by @johnkerl in #887
* Clean up file output handler error handling by @skitt in #886
* Use raw strings to avoid escapes by @skitt in #885
* Specify constant types except with iota by @skitt in #884
* Mlrval arrayval from []Mlrval to []*Mlrval by @johnkerl in #880
* Append slices directly instead of looping by @skitt in #879
* Fix mlrmap.Equals FieldCount comparison by @skitt in #878
* Ensure regression-test has a binary to test by @skitt in #877
* Avoid assuming ./mlr is the mlr to test by @skitt in #876
* Update release.yml by @jauderho in #867
* Update .goreleaser.yml by @jauderho in #866
* Goreleaser binary names by @johnkerl in #852
* Add CodeQL support by @jauderho in #838
New Contributors
* @vapniks made their first contribution in #854
* @Gary-Armstrong made their first contribution in #863
* @zachvalenta made their first contribution in #945
* @Poshi made their first contribution in #949
v6.0.0
======
This is a significant release with many improvements to user experience, documentation, and performance.
Please see What's new in Miller 6 for complete information.
https://miller.readthedocs.io/en/latest/new-in-miller-6
v5.10.3
=======
This release exists solely to resolve a Conda-build issue as discussed on #740. If you're not actively working on Conda packaging
for Miller, this release has no added value for you above 5.10.2.
Likewise, there's no Windows mlr.exe for this final (technical & specific) Miller 5.x release -- for Miller 6.0.0 (coming soon!)
and above there will be mlr.exe as a reliably standard part of each release.
Also note that the tarball is named miller-5.10.3.tar.gz, in contrast to mlr-5.10.2.tar.gz and likewise for all earlier releases.
This is being done for forward compatibility with Miller 6.0.0 and beyond which will use names of the form miller-6.0.0.tar.gz, as
proposed in #360.
miller: update to 5.10.4. Miller 5 is long-gone; 6.0.0 was released almost a year ago. Yet issue #1108 reports a critical memory-corruption bug on 5.10.3; this fixes that.
miller: update to 5.10.2. This release restore mlr manpage to distro file.
miller: update to 5.10.1. Various bugfixes. No upstream ChangeLog.
miller: update to 5.10.0. ChangeLog: Features: - The unsparsify -f feature - The new sort-within-records verb is an old ask, underway from the Go port, backported to C - Likewise the truncate DSL function Bugfixes: - The count -n feature was not implemented as intended - Pretty-print format now works correctly with --headerless-csv-output - The seqgen verb now correctly tracks NR and FNR in the records it emits - An intermittent JSON-parsing bug has been fixed
miller: update to 5.9.1. ChangeLog: Security update: disallow --prepipe in .mlrrc As of Miller 5.9.0, you can have a .mlrrc file containing preferred flags. As reported in #363, it would be possible for someone to prepare a repository or some other zipfile/tarfile, for example, containing datasets, and send it to you. They could have a line of the form prepipe do_something_bad; cat in that repository, so when you ran any mlr commands in there, it would run the do_something_bad command (whatever that might be). The fix is (a) disallow prepipe within .mlrrc files; (b) as a consolation, allow new prepipe-zcat and prepipe-gunzip options which are safe to use. Fixes CVE-2020-15167.
miller: update to 5.9.0. ChangeLog: - You can now save common defaults in a ~/.mlrrc. For example, if you normally process CSV files, you can say that in your ~/.mlrrc and you can leave off the --csv flag from your mlr commands. You can read more about this feature here, or in man mlr, or in mlr --help.
miller: update to 5.8.0.
ChangeLog:
Features
The new count verb is a keystroke-saver for stats1 -a count -f
{some field name}.
--jsonx and --ojsonx are keystroke-savers for --json --jvstack
and --ojson --jvstack, which is to say, multi-line pretty-printed
JSON format.
The new -s name=value feature for mlr put and mlr filter gives
you simpler access to environment variables in your Miller
script, as requested in #315.
Bugfixes
mlr format-values is no longer SEGVing on CSV/TSV input. This
was reported on #330.
#313 fixes a corner case when field names within command-line
arguments have embedded newlines.
Line/column indicators for JSON-formatting error messages are
now correct (previously they were showing up as 0).
end {print NF} no longer SEGVs. This was reported in #330.
Several broken doc links were fixed up as reported on #329.
miller: update to 5.7.0.
ChangeLog:
Features
The new remove-empty-columns and skip-trivial-records are
keystroke-savers for things which would other require DSL
syntax, as tracked in #274.
Bugfixes
A bug regarding optional regex-pattern groups was fixed in
#277.
As of #294 you can now specify --implicit-csv-header for the
join-file in mlr join.
A bug with spaces in XTAB-file values was fixed on #296.
A bug with missing final newline for XTAB-formatted files
using MMAP files was fixed on #301.
Documentation
Look-and-feel at http://johnkerl.org/miller/doc/ is (hopefully)
improved, including clearer visual indication of which section/page
you're currently looking at. Note that this change has been
live for a few weeks, as look-and-feel-related doc-mods from
post-5.6.2 were backported to http://johnkerl.org/miller/doc/.
#282 improves DSL-function documentation at
http://johnkerl.org/miller/doc/reference-dsl.html#Built-in_functions_for_filter_and_put,_summary
Note
Support for mmap mode has been entirely discontinued. This is an
invisible change and should not affect you at all. For anyone
interested in lower-level details, though, the summary is as follows:
For an incremental performance gain (perhaps 10-20% run time
at most, but see below), within the C source code one can use
the mmap system call to access input files via pointer arithmetic
rather than malloc-and-memcopy using stdio.
However mmap is not available when reading from standard
input -- it cannot be memory-mapped.
This means all file-format readers are implemented twice
within the Miller source code.
While I try to regression-test Miller thoroughly, running
all canned tests through mmap and stdio mode, I've nonetheless
found my mmap implementations liable to corner-cases which I
miss but users find: for example #29, #102, and #296.
As tracked on #160, various operating systems do not release
mmapped pages after use as one might intuit, meaning that for
large files and/or large numbers of files, I've for a long time
now needed to have Miller opt out of mmap usage for precisely
those cases which most need the performance gain: see #160,
#181, and #256.
Additionally, mmap is not used at all for Windows/MSYS2 so
there is nothing to lose there.
For these reasons, keeping mmap mode isn't worth the development
overhead.
As of release 5.6.3, the mlr executable will still accept the --mmap
and --no-mmap command-line flags as no-ops, for backward compatibility.
The caveat for you is that for everyday small files, the default
was previously mmap mode and is now stdio (except mlr ... < filename
or ... | mlr ... which have always used stdio). There is the off
chance that this will newly reveal an old, latent bug or two
somewhere.
I've re-run regressions in valgrind mode to aggressively catch any
errors, but, please let me know ASAP via GitHub issue of any
unexpected behavior in 5.7.0.
miller: update to 5.6.2.
ChangeLog:
v5.6.2
Bug fixes:
#271 fixes a corner-case bug with more than 100 CSV/TSV files with
headers of varying lengths.
Documentation:
The new http://johnkerl.org/miller/doc/whyc-details.html is an
elaboration on http://johnkerl.org/miller/doc/whyc.html which answers
a question posed by @BurntSushi on Reddit a couple years ago which
I did not address in detail at the time.
v5.6.1
The only change is that http://johnkerl.org/miller/doc is now
more mobile-friendly. All build artifacts are the same as at
https://github.com/johnkerl/miller/releases/tag/v5.6.0
v5.6.0
The new system DSL function allows you to run arbitrary shell commands
and store them in field values. Some example usages are documented
here. This is in response to issues #246 and #209.
There is now support for ASV and USV file formats. This is in response
to issue #245.
The new format-values verb allows you to apply numerical formatting
across all record values. This is in response to issue #252.
Documentation:
The new DKVP I/O in Python sample code now works for Python 2 as
well as Python 3.
There is a new cookbook entry on doing multiple joins. This is in
response to issue #235.
Bugfixes:
The toupper, tolower, and capitalize DSL functions
are now UTF-8 aware, thanks to @sheredom's marvelous
https://github.com/sheredom/utf8.h. The internationalization page
has also been expanded. This is in response to issue #254.
#250 fixes a bug using in-place mode in conjunction with verbs
(such as rename or sort) which take field-name lists as arguments.
#253 fixes a bug in the label when one or more names are common
between old and new.
#251 fixes a corner-case bug when (a) input is CSV; (b) the last
field ends with a comma and no newline; (c) input is from standard
input and/or --no-mmap is supplied.
v5.5.0
The new positional-indexing feature resolves #236 from @aborruso. You
can now get the name of the 3rd field of each record via $[[3]], and
its value by $[[[3]]]. These are both usable on either the left-hand
or right-hand side of assignment statements, so you can more easily
do things like renaming fields progrmatically within the DSL.
There is a new capitalize DSL function, complementing the
already-existing toupper. This stems from #236.
There is a new skip-trivial-records verb, resolving #197. Similarly,
there is a new remove-empty-columns verb, resolving #206. Both are
useful for data-cleaning use-cases.
Another pair is #181 and #256. While Miller uses mmap internally
(and invisibily) to get approximately a 20% performance boost over
not using it, this can cause out-of-memory issues with reading either
large files, or too many small ones. Now, Miller automatically avoids
mmap in these cases. You can still use --mmap or --no-mmap if you
want manual control of this.
There is a new --ivar option for the nest verb which complements
the already-existing --evar. This is from #260 thanks to @jgreely.
There is a new keystroke-saving urandrange DSL function:
urandrange(low, high) is the same as low + (high - low) *
urand(). This arose from #243.
There is a new -v option for the cat verb which writes a low-level
record-structure dump to standard error.
There is a new -N option for mlr which is a keystroke-saver for
--implicit-csv-header --headerless-csv-output.
Documentation:
The new FAQ entry
http://johnkerl.org/miller/doc/faq.html#How_to_escape_'%3F'_in_regexes%3F
resolves #203.
The new FAQ entry
http://johnkerl.org/miller/doc/faq.html#How_can_I_filter_by_date%3F
resolves #208.
#244 fixes a documentation issue while highlighting the need for #241.
Bugfixes:
There was a SEGV using nest within then-chains, fixed in response
to #220.
Quotes and backslashes weren't being escaped in JSON output with
--jvquoteall; reported on #222.
v5.4.0
The new clean-whitespace verb resolves #190 from @aborruso. Along with
the new functions strip, lstrip, rstrip, collapse_whitespace, and
clean_whitespace, there is now both coarse-grained and fine-grained
control over whitespace within field names and/or values. See the
linked-to documentation for examples.
The new altkv verb resolves #184 which was originally opened via an
email request. This supports mapping value-lists such as a,b,c,d to
alternating key-value pairs such as a=b,c=d.
The new fill-down verb resolves #189 by @aborruso. See the linked-to
documentation for examples.
The uniq verb now has a uniq -a which resolves #168 from @sjackman.
The new regextract and regextract_or_else functions resolve #183
by @aborruso.
The new ssub function arises from #171 by @dohse, as a simplified way
to avoid escaping characters which are special to regular-expression
parsers.
There are new localtime functions in response to #170 by
@sitaramc. However note that as discussed on #170 these do
not undo one another in all circumstances. This is a non-issue
for timezones which do not do DST. Otherwise, please use with
disclaimers: localdate, localtime2sec, sec2localdate, sec2localtime,
strftime_local, and strptime_local.
Builds:
Windows build-artifacts are now available in Appveyor at
https://ci.appveyor.com/project/johnkerl/miller/build/artifacts,
and will be attached to this and future releases. This resolves #167,
#148, and #109.
Travis builds at https://travis-ci.org/johnkerl/miller/builds now
run on OSX as well as Linux.
An Ubuntu 17 build issue was fixed by @singalen on #164.
Documentation:
put/filter documentation was confusing as reported by @NikosAlexandris
on #169.
The new FAQ entry
http://johnkerl.org/miller-releases/miller-head/doc/faq.html#How_to_rectangularize_after_joins_with_unpaired?
resolves #193 by @aborruso.
The new cookbook entry
http://johnkerl.org/miller/doc/cookbook.html#Options_for_dealing_with_duplicate_rows
arises from #168 from @sjackman.
The unsparsify documentation had some words missing as reported by
@tst2005 on #194.
There was a typo in the cookpage page
http://johnkerl.org/miller/doc/cookbook.html#Full_field_renames_and_reassigns
as fixed by @tst2005 in #192.
Bugfixes:
There was a memory leak for TSV-format files only as reported by
@treynr on #181.
Dollar sign in regular expressions were not being escaped properly
as reported by @dohse on #171.
v5.3.0
Comment strings in data files: mlr --skip-comments allows
you to filter out input lines starting with #, for all file
formats. Likewise, mlr --skip-comments-with X lets you specify
the comment-string X. Comments are only supported at start of data
line. mlr --pass-comments and mlr --pass-comments-with X allow you
to forward comments to program output as they are read.
The count-similar verb lets you compute cluster sizes by cluster
labels.
While Miller DSL arithmetic gracefully overflows from 64-integer
to double-precision float (see also here), there are now the
integer-preserving arithmetic operators .+ .- .* ./ .// for those
times when you want integer overflow.
There is a new bitcount function: for example, echo x=0xf0000206 |
mlr put '$y=bitcount($x)' produces x=0xf0000206,y=7.
Issue 158: mlr -T is an alias for --nidx --fs tab, and mlr -t is an
alias for mlr --tsvlite.
The mathematical constants π and e have been renamed from PI and
E to M_PI and M_E, respectively. (It's annoying to get a syntax
error when you try to define a variable named E in the DSL, when
A through D work just fine.) This is a backward incompatibility,
but not enough of us to justify calling this release Miller 6.0.0.
Documentation:
As noted here, while Miller has its own DSL there will always be
things better expressible in a general-purpose language. The new page
Sharing data with other languages shows how to seamlessly share data
back and forth between Miller, Ruby, and Python. SQL-input examples
and SQL-output examples contain detailed information the interplay
between Miller and SQL.
Issue 150 raised a question about suppressing numeric conversion. This
resulted in a new FAQ entry How do I suppress numeric conversion?,
as well as the longer-term follow-on issue 151 which will make
numeric conversion happen on a just-in-time basis.
To my surprise, csvlite format options weren’t listed in mlr --help
or the manpage. This has been fixed.
Documentation for auxiliary commands has been expanded, including
within the manpage.
Bugfixes:
Issue 159 fixes regex-match of literal dot.
Issue 160 fixes out-of-memory cases for huge files. This is an old
bug, as old as Miller, and is due to inadequate testing of huge-file
cases. The problem is simple: Miller prefers memory-mapped I/O
(using mmap) over stdio since mmap is fractionally faster. Yet as
any processing (even mlr cat) steps through an input file, more and
more pages are faulted in -- and, unfortunately, previous pages are
not paged out once memory pressure increases. (This despite gallant
attempts with madvise.) Once all processing is done, the memory is
released; there is no leak per se. But the Miller process can crash
before the entire file is read. The solution is equally simple: to
prefer stdio over mmap for files over 4GB in size. (This 4GB threshold
is tunable via the --mmap-below flag as described in the manpage.)
Issue 161 fixes a CSV-parse error (with error message "unwrapped
double quote at line 0") when a CSV file starts with the UTF-8
byte-order-mark ("BOM") sequence 0xef 0xbb 0xbf and the header line
has double-quoted fields. (Release 5.2.0 introduced handling for
UTF-8 BOMs, but missed the case of double-quoted header line.)
Issue 162 fixes a corner case doing multi-emit of aggregate variables
when the first variable name is a typo.
The Miller JSON parser used to error with Unable to parse JSON data:
Line 1 column 0: Unexpected 0x00 when seeking value on empty input,
or input with trailing whitespace; this has been fixed.
miller: Add flex as tool dependency (flex is explicitly needed in c/parsing/Makefile for mlr_dsl_lexer.c.)
Updated miller to 5.2.2. 5.2.2 This bugfix release delivers a fix for #147 where a memory allocation failed beyond 4GB. 5.2.1 Fix non-x86/gcc7 build error
Switch github HOMEPAGEs to https.
Updated miller to 5.2.0.
This release contains mostly feature requests.
Features:
The stats1 verb now lets you use regular expressions to specify
which field names to compute statistics on, and/or which to
group by. Full details are here.
The min and max DSL functions, and the min/max/percentile
aggregators for the stats1 and merge-fields verbs, now support
numeric as well as string field values. (For mixed string/numeric
fields, numbers compare before strings.) This means in particular
that order statistics -- min, max, and non-interpolated percentiles
-- as well as mode, antimode, and count are now possible on
string-only (or mixed) fields. (Of course, any operations
requiring arithmetic on values, such as computing sums, averages,
or interpolated percentiles, yield an error on string-valued
input.)
There is a new DSL function mapexcept which returns a copy of
the argument with specified key(s), if any, unset. The motivating
use-case is to split records to multiple filenames depending
on particular field value, which is omitted from the output:
mlr --from f.dat put 'tee > "/tmp/data-".$a, mapexcept($*, "a")'
Likewise, mapselect returns a copy of the argument with only
specified key(s), if any, set. This resolves #137.
A new -u option for count-distinct allows unlashed counts for
multiple field names. For example, with -f a,b and without -u,
count-distinct computes counts for distinct pairs of a and b
field values. With -f a,b and with -u, it computes counts for
distinct a field values and counts for distinct b field values
separately.
If you build from source, you can now do ./configure without
first doing autoreconf -fiv. This resolves #131.
The UTF-8 BOM sequence 0xef 0xbb 0xbf is now automatically
ignored from the start of CSV files. (The same is already done
for JSON files.) This resolves #138.
For put and filter with -S, program literals such as the 6 in
$x = 6 were being parsed as strings. This is not sensible, since
the -S option for put and filter is intended to suppress numeric
conversion of record data, not program literals. To get string
6 one may use $x = "6".
Documentation:
A new cookbook example shows how to compute differences between
successive queries, e.g. to find out what changed in time-varying
data when you run and rerun a SQL query.
Another new cookbook example shows how to compute interquartile
ranges.
A third new cookbook example shows how to compute weighted
means.
Bugfixes:
CRLF line-endings were not being correctly autodetected when
I/O formats were specified using --c2j et al.
Integer division by zero was causing a fatal runtime exception,
rather than computing inf or nan as in the floating-point case.
Updated miller to 5.1.0.
This is a relatively minor release of Miller, containing feature
requests and bugfixes while I've been working on the Windows port
(which is nearly complete).
Features:
JSON arrays: as described here, Miller being a tabular data
processor isn't well-position to handle arbitrary JSON. (See
jq for that.) But as of 5.1.0, arrays are converted to maps
with integer keys, which are then at least processable using
Miller. Details are here. The short of it is that you now have
three options for the main mlr executable:
--json-map-arrays-on-input Convert JSON array indices to Miller
map keys. (This is the default.) --json-skip-arrays-on-input
Disregard JSON arrays. --json-fatal-arrays-on-input Raise a fatal
error when JSON arrays are encountered in the input.
This resolves #133.
The new mlr fraction verb makes possible in a few keystrokes
what was only possible before using two-pass DSL logic: here
you can turn numerical values down a column into their
fractional/percentage contribution to column totals, optionally
grouped by other key columns.
The DSL functions strptime and strftime now handle fractional
seconds. For parsing, use %S format as always; for formatting,
there are now %1S through %9S which allow you to configure a
specified number of decimal places. The return value from
strptime is now floating-point, not integer, which is a minor
backward incompatibility not worth labeling this release as
6.0.0. (You can work around this using int(strptime(...)).) The
DSL functions gmt2sec and sec2gmt, which are keystroke-savers
for strptime and strftime, are similarly modified, as is the
sec2gmt verb. This resolves #125.
A few nearly-standalone programs -- which do not have anything
to do with record streams -- are packaged within the Miller.
(For example, hex-dump, unhex, and show-line-endings commands.)
These are described here.
The stats1 and merge-fields verbs now support an antimode
aggregator, in addition to the existing mode aggregator.
The join verb now by default does not require sorted input,
which is the more common use case. (Memory-parsimonious joins
which require sorted input, while no longer the default, are
available using -s.) This another minor backward incompatibility
not worth making a 6.0.0 over. This resolves #134.
mlr nest has a keystroke-saving --evar option for a common use
case, namely, exploding a field by value across records.
Documentation:
The DSL reference now has per-function descriptions.
There is a new feature-counting example in the cookbook.
Bugfixes:
mlr join -j -l was not functioning correctly. This resolves
#136.
JSON escapes on output (\t and so on) were incorrect. This
resolves #135.
Updated miller to 5.0.1.
Two minor bugfixes
As described in #132, mlr nest was incorrectly splitting fields
with multi-character separators.
The XTAB-format reader, when using multi-character IPS, was
incorrectly splitting key-value pairs, but only when reading
from standard input (e.g. on a pipe or less-than redirect).
Updated miller to 5.0.0.
Autodetected line-endings, in-place mode, user-defined functions, and more
This major release significantly expands the expressiveness of the DSL for mlr put and mlr filter. (The upcoming 5.1.0 release will add the ability to aggregate across all columns for non-DSL verbs such as mlr stats1 and mlr stats2. As well, a Windows port is underway.)
Please also see the Miller main docs.
Simple but impactful features:
Line endings (CRLF vs. LF, Windows-style vs. Unix-style) are now autodetected. For example, files (including CSV) with LF input will lead to LF output unless you specify otherwise.
There is now an in-place mode using mlr -I.
Major DSL features:
You can now define your own functions and subroutines: e.g. func f(x, y) { return x**2 + y**2 }.
New local variables are completely analogous to out-of-stream variables: sum retains its value for the duration of the expression it's defined in; @sum retains its value across all records in the record stream.
Local variables, function parameters, and function return types may be defined untyped or typed as in x = 1 or int x = 1, respectively. There are also expression-inline type-assertions available. Type-checking is up to you: omit it if you want flexibility with heterogeneous data; use it if you want to help catch misspellings in your DSL code or unexpected irregularities in your input data.
There are now four kinds of maps. Out-of-stream variables have always been scalars, maps, or multi-level maps: @a=1, @b[1]=2, @c[1][2]=3. The same is now true for local variables, which are new to 5.0.0. Stream records have always been single-level maps; $* is a map. And as of 5.0.0 there are now map literals, e.g. {"a":1, "b":2}, which can be defined using JSON-like syntax (with either string or integer keys) and which can be nested arbitrarily deeply.
You can loop over maps -- $*, out-of-stream variables, local variables, map-literals, and map-valued function return values -- using for (k, v in ...) or the new for (k in ...) (discussed next). All flavors of map may also be used in emit and dump statements.
User-defined functions and subroutines may take map-valued arguments, and may return map values.
Some built-in functions now accept map-valued input: typeof, length, depth, leafcount, haskey. There are built-in functions producing map-valued output: mapsum and mapdiff. There are now string-to-map and map-to-string functions: splitnv, splitkv, splitnvx, splitkvx, joink, joinv, and joinkv.
Minor DSL features:
For iterating over maps (namely, local variables, out-of-stream variables, stream records, map literals, or return values from map-valued functions) there is now a key-only for-loop syntax: e.g. for (k in $*) { ... }. This is in addition to the already-existing for (k, v in ...) syntax.
There are now triple-statement for-loops (familiar from many other languages), e.g. for (int i = 0; i < 10; i += 1) { ... }.
mlr put and mlr filter now accept multiple -f for script files, freely intermixable with -e for expressions. The suggested use case is putting user-defined functions in script files and one-liners calling them using -e. Example: myfuncs.mlr defines the function f(...), then mlr put -f myfuncs.mlr -e '$o = f($i)' myfile.dat. More information is here.
mlr filter is now almost identical to mlr put: it can have multiple statements, it can use begin and/or end blocks, it can define and invoke functions. Its final expression must evaluate to boolean which is used as the filter criterion. More details are here.
The min and max functions are now variadic: $o = max($a, $b, $c).
There is now a substr function.
While ENV has long provided read-access to environment variables on the right-hand side of assignments (as a getenv), it now can be at the left-hand side of assignments (as a putenv). This is useful for subsidiary processes created by tee, emit, dump, or print when writing to a pipe.
Handling for the # in comments is now handled in the lexer, so you can now (correctly) include # in strings.
Separators are now available as read-only variables in the DSL: IPS, IFS, IRS, OPS, OFS, ORS. These are particularly useful with the split and join functions: e.g. with mlr --ifs tab ..., the IFS variable within a DSL expression will evaluate to a string containing a tab character.
Syntax errors in DSL expressions now have a little more context.
DSL parsing and execution are a bit more transparent. There have long been -v and -t options to mlr put and mlr filter, which print the expression's abstract syntax tree and do a low-level parser trace, respectively. There are now additionally -a which traces stack-variable allocation and -T which traces statements line by line as they execute. While -v, -t, and -a are most useful for development of Miller, the -T option gives you more visibility into what your Miller scripts are doing. See also here.
Verbs:
most-frequent and least-frequent as requested in #110.
seqgen makes it easy to generate data from within Miller: please also see here for a usage example.
unsparsify makes it easy to rectangularize data where not all records have the same fields.
cat -n now takes a group-by (-g) option, making it easy to number records within categories.
count-distinct,
uniq,
most-frequent,
least-frequent,
top, and
histogram
now take a -o option for specifying their output field names, as requested in #122.
Median is now a synonym for p50 in stats1.
You can now start a then chain with an initial then, which is nice in backslashy/multiline-continuation contexts.
This was requested in #130.
I/O options:
The print statement may now be used with no arguments, which prints a newline, and a no-argument printn prints nothing but creates a zero-length file in redirected-output context.
Pretty-print format now has a --pprint --barred option (for output only, not input). For an example, please see here.
There are now keystroke-savers of the form --c2p which abbreviate --icsvlite --opprint, and so on.
Miller's map literals are JSON-looking but allow integer keys which JSON doesn't. The
--jknquoteint and --jvquoteall flags for mlr (when using JSON output) and mlr put (for dump) provide control over double-quoting behavior.
Documents new since the previous release:
Miller in 10 minutes is a long-overdue addition: while Miller's detailed documentation is evident, there has been a lack of more succinct examples.
The cookbook has likewise been expanded, and has been split out
into three parts: part 1, part
2, part 3.
A bit more background on C performance compared to other languages I experimented with, early on in the development of Miller, is here.
On-line help:
Help for DSL built-in functions, DSL keywords, and verbs is accessible using mlr -f, mlr -k, and mlr -l respectively; name-only lists are available with mlr -F, mlr -K, and mlr -L.
Bugfixes:
A corner-case bug causing a segmentation violation on two sub/gsub statements within a single put, the first one matching its pattern and the second one not matching its pattern, has been fixed.
Backward incompatibilities: This is Miller 5.0.0, not 4.6.0, due to the following (all relatively minor):
The v variables bound in for-loops such as for (k, v in some_multi_level_map) { ... } can now be map-valued if the v specifies a non-terminal in the map.
There are new keywords such as var, int, float, num, str, bool, map, IPS, IFS, IRS, OPS, OFS, ORS which can no longer be used as variable names. See mlr -k for the complete list.
Unset of the last key in an map-valued variable's map level no longer removes the level: e.g. with @v[1][2]=3 and unset @v[1][2] the @v variable would be empty. As of 5.0.0, @v has key 1 with an empty-map value.
There is no longer type-inference on literals: "3"+4 no longer gives 7. (That was never a good idea.)
The typeof function used to say things like MT_STRING; now it says things like string.
Updated miller to 4.5.0.
4.5.0
Customizable output format for redirected output
In a natural follow-on to the 4.4.0 redirected-output feature, the
4.5.0 release allows your tap-files to be in a different output
format from the main program output.
For example, using
mlr --icsv --opprint ... then put --ojson 'tee > "mytap-".$a.".dat",
$*' then ...
the input is CSV, the output is pretty-print tabular, but the
tee-files output is written in JSON format. Likewise --ofs, --ors,
--ops, --jvstack, and all other output-formatting options from the
main help at mlr -h and/or man mlr default to the main command-line
options, and may be overridden with flags supplied to mlr put and
mlr tee.
4.4.0
Redirected output, row-value shift, and other features
The principal feature of Miller 4.4.0 is redirected output. Inspired
by awk, Miller lets you tap/tee your data as it's processed, run
output through subordinate processes such as gzip and jq, split a
single file into multiple files per an account-ID column, and so
on.
Details:
http://johnkerl.org/miller/doc/reference.html#Redirected-output_statements_for_put
Other features:
mlr step -a shift allows you to place the previous record's
values alongside the current record's values:
http://johnkerl.org/miller/doc/reference.html#step
mlr head, when used without the group-by flag (-g), stops after
the specified number of records has been output. For example,
even with a multi-gigabyte data file, mlr head -n 10 hugefile.dat
will complete quickly after producing the first ten records
from the file.
The sec2gmtdate verb, and sec2gmtdate function for filter/put,
is new: please see
http://johnkerl.org/miller/doc/reference.html#sec2gmtdate and
http://johnkerl.org/miller/doc/reference.html#Functions_for_filter_and_put.
sec2gmt and sec2gmtdate both leave non-numbers as-is, rather
than formatting them as (error). This is particularly relevant
for formatting nullable epoch-seconds columns in SQL-table
output: if a column value is NULL then after sec2gmt or
sec2gmtdate it will still be NULL.
The dot operator has been universalized to work with any data
type and produce a string. For example, if the field n has
integers, then instead of typing mlr put '$name = "value:".string($n)'
you can now simply domlr put '$name = "value:".$n'. This is
particularly timely for creating filenames for redirected
print/dump/tee/emit output.
The online documents now have a copy of the Miller manpage:
http://johnkerl.org/miller/doc/manpage.html
Bugfix: inside filter/put, $x=="" was distinct from isempty($x).
This was nonsensical; now both are the same.
Updated textproc/miller 3.4.0 to 4.3.0
--------------------------------------
From: https://github.com/johnkerl/miller/releases
Summary:
v4.3.0
Interpolated percentiles, markdown-tabular output format, CSV-quote preservation
v4.2.0
Multi-emit
v4.1.0
for/if/while and various features
v4.0.0
Variables, begin/end blocks, pattern-action blocks
v3.5.0
New data-rearrangers: nest, shuffle, repeat; misc. features
Update miller to 3.4.0.
Use release tarball and drop autotools dependencies.
Changes in 3.4.0:
JSON, reshape, regex captures, and more
Primary features:
JSON is now a supported format for input and output. Miller handles tabular data, and JSON supports arbitrarily deeply nested data structures, so if you want general JSON processing you should use jq. But if you have tabular data represented in JSON then Miller can now handle that for you. Please see the reference page and the FAQ.
Reshape is a standard data-processing idiom, now available in Miller: http://johnkerl.org/miller/doc/reference.html#reshape
Incidentally (not part of this release, but new since the last release) Miller is now available in FreeBSD's package manager: https://www.freshports.org/textproc/miller/. A full list of distributions containing Miller may be found here.
Miller is not yet available from within Fedora/CentOS, but as a step toward this goal, an SRPM is included in this release (see file-list below).
DSL enhancements for mlr put and mlr filter:
Regex captures \0 through \9: http://johnkerl.org/miller/doc/reference.html#Regex_captures
Ternary operator in expression right-hand sides: e.g. mlr put '$y = $x < 0.5 ? 0 : 1'
Boolean literals true and false
Final semicolon is now allowed: e.g. mlr put '$x=1;$y=2;'
Environment variables are now accessible, where environment-variable names may be string literals or arbitrary expressions: mlr put '$home = ENV["HOME"]' or mlr put '$value = ENV[$name]'.
While records are still string-to-string maps for input and output, and between then statements, types are preserved between multiple statements within a put. Example: mlr put '$y = string($x); $z = $y . $y' works as expected, without requring mlr put '$y = string($x); $z = string($y) . string($y)' as before.
Bug fixes:
Mixed-format join, e.g. CSV file joined with DKVP file, was incorrectly computing default separators (IRS, IFS, IPS). This resulted in records not being joined together.
Segmentation violation on non-standard-input read of files with size an exact multiple of page size and not ending in IRS, e.g. newline. (This is less of a corner case than it sounds: for example, leave a long-running program running with output redirected to a file, then in a sleep-and-process loop, have Miller process that file. The former program's stdio library will likely be doing block-sized buffered I/O, where block sizes will often be multiples of system page size and the block will almost surely not ending a newline.)
Acknowledgements: Big thank-yous to @gregfr and @aaronwolen for feature requests including reshape and regex captures, and to @jungle-boogie for his work getting Miller into FreeBSD. Also, ongoing thanks to @0-wiz-0 for his past work on configure support, making it possible for Miller to be put to use in multiple operating systems.
3.3.2
Bootstrap sampling, EWMA, merge-fields, isnull/isnotnull functions
@johnkerl johnkerl released this on Jan 11 · 497 commits to master since this release
Bootstrap sampling in mlr bootstrap: http://johnkerl.org/miller/doc/reference.html#bootstrap. Compare to reservoir sampling in mlr sample: http://johnkerl.org/miller/doc/reference.html#sample.
Exponentially weighted moving averages in mlr step -a ewma: principally useful for smoothing of noisy time series, e.g. finely sampled system-resource utilization to give one of many possible examples. Please see http://johnkerl.org/miller/doc/reference.html#step.
"Horizontal" univariate statistics in mlr merge-fields, compared to mlr stats which is "vertical". Also allows collapsing multiple fields into one, such as in_bytes and out_bytes data fields summing to bytes_sum. This can also be done easily using mlr put. However, mlr merge-fields allows aggregation of more than just a pair of field names, and supports pattern-matching on field names. Please see http://johnkerl.org/miller/doc/reference.html#merge-fields for more information.
isnull and isnotnull functions for mlr filter and mlr put.
stats1, stats2, merge-fields, step, and top correctly handle not only missing fields (in the row-heterogeneous-data case) but also null-valued fields.
Minor memory-management improvements.
Update miller to 3.2.2: Many changes; speed ups, autoconf support, ....
Update textproc/miller to 2.2.0: Multi-character RS,FS,PS You can process CRLF-terminated DKVP files with mlr --dkvp --rs crlf. You can process LF-terminated CSV files with mlr --csv --rs lf. You can process TSV using mlr --fs tab; you can convert TSV to CSV using mlr --ifs tab --ofs comma. Along with many more possibilities. Please see mlr -h for more information. There is one minor, backward-incompatible change which I felt not worth calling this 3.0.0: default field separator for NIDX format is now space, not comma.
Update 2.1.1 to 2.1.3 --------------------- ChangeLog is unknown.
Update miller to 2.1.1. Changes: v2.1.1 Incremental read-performance increase for CSV format While #51 is still underway, already there is nearly a 2x read-performance increase in v2.1.1 over v2.1.0. v2.1.0 Minor enhancements and bug fixes Highlights: travis-CI integration (thanks @SikhNerd!); hour-minute-second functions; fixed pretty-print alignment of UTF-8 data.
Import miller-2.0.0 as textproc/miller. Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV. With Miller, you get to use named fields without needing to count positional indices. This is something the Unix toolkit always could have done, and arguably always should have done. It operates on key-value-pair data while the familiar Unix tools operate on integer-indexed fields: if the natural data structure for the latter is the array, then Miller's natural data structure is the insertion-ordered hash map. This encompasses a variety of data formats, including but not limited to the familiar CSV. (Miller can handle positionally-indexed data as a special case.)