==> Building on reshiram ==> Attempt to run pkgver()... ==> Checking for remote environment... ==> Syncing package to remote host... sending incremental file list ./ PKGBUILD 1,399 57% 0.00kB/s 0:00:00 2,430 100% 1006.84kB/s 0:00:00 (xfr#1, to-chk=5/7) config.yaml 38 100% 37.11kB/s 0:00:00 38 100% 37.11kB/s 0:00:00 (xfr#2, to-chk=4/7) mimir-2.6.0-1.log 247 100% 241.21kB/s 0:00:00 247 100% 241.21kB/s 0:00:00 (xfr#3, to-chk=3/7) systemd.service 700 69% 683.59kB/s 0:00:00 1,011 100% 987.30kB/s 0:00:00 (xfr#4, to-chk=2/7) sysusers.conf 45 100% 43.95kB/s 0:00:00 45 100% 43.95kB/s 0:00:00 (xfr#5, to-chk=1/7) tmpfiles.conf 34 100% 33.20kB/s 0:00:00 34 100% 33.20kB/s 0:00:00 (xfr#6, to-chk=0/7) sent 1,416 bytes received 187 bytes 3,206.00 bytes/sec total size is 3,805 speedup is 2.37 ==> Patching arch to riscv64... ==> Running extra-riscv64-build -- -d /home/felix/packages/riscv64-pkg-cache:/var/cache/pacman/pkg -l root3 on remote host... [?25l:: Synchronizing package databases... core downloading... extra downloading... community downloading... :: Starting full system upgrade... there is nothing to do [?25h==> Building in chroot for [extra] (riscv64)... ==> Synchronizing chroot copy [/var/lib/archbuild/extra-riscv64/root] -> [root3]...done ==> Making package: mimir 2.6.0-1 (Tue Mar 7 06:25:14 2023) ==> Retrieving sources...  -> Updating mimir git repo... From https://github.com/grafana/mimir - [deleted] (none) -> 56quarters/2.6-removal - [deleted] (none) -> aasp/minio-update - [deleted] (none) -> add-default-am-config - [deleted] (none) -> add-otlp-conversion-docs - [deleted] (none) -> aldernero/backfill_validation - [deleted] (none) -> aldernero/cleanup-of-partial-blocks - [deleted] (none) -> aldernero/delete-partial-blocks - [deleted] (none) -> dimitar/copy-migrate-from-cortex - [deleted] (none) -> dimitar/remove-jsonnet-content-from-migrate-from-cortex - [deleted] (none) -> dimitar/st-gw-shortcut-queries-with-0-series-labelvalues - [deleted] (none) -> dimitar/update-mimirtool-confgig-convert-descriptors - [deleted] (none) -> lamida/admin-cache - [deleted] (none) -> leizor/tsdb-chunks - [deleted] (none) -> leizor/tsdb-index-health - [deleted] (none) -> leizor/tsdb-print-chunk - [deleted] (none) -> mark_series_ephemeral_in_distributor - [deleted] (none) -> read-write-deployment/docs - [deleted] (none) -> refactor/compactor - [deleted] (none) -> snyk-monitor-workflow - [deleted] (none) -> support-multiple-querier-scaled-objects - [deleted] (none) -> update-otel-col-to-latest-release - [deleted] (none) -> refs/pull/1679/merge - [deleted] (none) -> refs/pull/1879/merge - [deleted] (none) -> refs/pull/2606/merge - [deleted] (none) -> refs/pull/2652/merge - [deleted] (none) -> refs/pull/2831/merge - [deleted] (none) -> refs/pull/3198/merge - [deleted] (none) -> refs/pull/3215/merge - [deleted] (none) -> refs/pull/3243/merge - [deleted] (none) -> refs/pull/3411/merge - [deleted] (none) -> refs/pull/3541/merge - [deleted] (none) -> refs/pull/3554/merge - [deleted] (none) -> refs/pull/3629/merge - [deleted] (none) -> refs/pull/3637/merge - [deleted] (none) -> refs/pull/3740/merge - [deleted] (none) -> refs/pull/3784/merge - [deleted] (none) -> refs/pull/3798/merge - [deleted] (none) -> refs/pull/3852/merge - [deleted] (none) -> refs/pull/3866/merge - [deleted] (none) -> refs/pull/3872/merge - [deleted] (none) -> refs/pull/3897/merge - [deleted] (none) -> refs/pull/3911/merge - [deleted] (none) -> refs/pull/3929/merge - [deleted] (none) -> refs/pull/3930/merge - [deleted] (none) -> refs/pull/3942/merge - [deleted] (none) -> refs/pull/3946/merge - [deleted] (none) -> refs/pull/3948/merge - [deleted] (none) -> refs/pull/3950/merge - [deleted] (none) -> refs/pull/3951/merge - [deleted] (none) -> refs/pull/3952/merge - [deleted] (none) -> refs/pull/3956/merge - [deleted] (none) -> refs/pull/3962/merge - [deleted] (none) -> refs/pull/3968/merge - [deleted] (none) -> refs/pull/3978/merge - [deleted] (none) -> refs/pull/3982/merge remote: Enumerating objects: 16745, done. remote: Counting objects: 0% (1/9200) remote: Counting objects: 1% (92/9200) remote: Counting objects: 2% (184/9200) remote: Counting objects: 3% (276/9200) remote: Counting objects: 4% (368/9200) remote: Counting objects: 5% (460/9200) remote: Counting objects: 6% (552/9200) remote: Counting objects: 7% (644/9200) remote: Counting objects: 8% (736/9200) remote: Counting objects: 9% (828/9200) remote: Counting objects: 10% (920/9200) remote: Counting objects: 11% (1012/9200) remote: Counting objects: 12% (1104/9200) remote: Counting objects: 13% (1196/9200) remote: Counting objects: 14% (1288/9200) remote: Counting objects: 15% (1380/9200) remote: Counting objects: 16% (1472/9200) remote: Counting objects: 17% (1564/9200) remote: Counting objects: 18% (1656/9200) remote: Counting objects: 19% (1748/9200) remote: Counting objects: 20% (1840/9200) remote: Counting objects: 21% (1932/9200) remote: Counting objects: 22% (2024/9200) remote: Counting objects: 23% (2116/9200) remote: Counting objects: 24% (2208/9200) remote: Counting objects: 25% (2300/9200) remote: Counting objects: 26% (2392/9200) remote: Counting objects: 27% (2484/9200) remote: Counting objects: 28% (2576/9200) remote: Counting objects: 29% (2668/9200) remote: Counting objects: 30% (2760/9200) remote: Counting objects: 31% (2852/9200) remote: Counting objects: 32% (2944/9200) remote: Counting objects: 33% (3036/9200) remote: Counting objects: 34% (3128/9200) remote: Counting objects: 35% (3220/9200) remote: Counting objects: 36% (3312/9200) remote: Counting objects: 37% (3404/9200) remote: Counting objects: 38% (3496/9200) remote: Counting objects: 39% (3588/9200) remote: Counting objects: 40% (3680/9200) remote: Counting objects: 41% (3772/9200) remote: Counting objects: 42% (3864/9200) remote: Counting objects: 43% (3956/9200) remote: Counting objects: 44% (4048/9200) remote: Counting objects: 45% (4140/9200) remote: Counting objects: 46% (4232/9200) remote: Counting objects: 47% (4324/9200) remote: Counting objects: 48% (4416/9200) remote: Counting objects: 49% (4508/9200) remote: Counting objects: 50% (4600/9200) remote: Counting objects: 51% (4692/9200) remote: Counting objects: 52% (4784/9200) remote: Counting objects: 53% (4876/9200) remote: Counting objects: 54% (4968/9200) remote: Counting objects: 55% (5060/9200) remote: Counting objects: 56% (5152/9200) remote: Counting objects: 57% (5244/9200) remote: Counting objects: 58% (5336/9200) remote: Counting objects: 59% (5428/9200) remote: Counting objects: 60% (5520/9200) remote: Counting objects: 61% (5612/9200) remote: Counting objects: 62% (5704/9200) remote: Counting objects: 63% (5796/9200) remote: Counting objects: 64% (5888/9200) remote: Counting objects: 65% (5980/9200) remote: Counting objects: 66% (6072/9200) remote: Counting objects: 67% (6164/9200) remote: Counting objects: 68% (6256/9200) remote: Counting objects: 69% (6348/9200) remote: Counting objects: 70% (6440/9200) remote: Counting objects: 71% (6532/9200) remote: Counting objects: 72% (6624/9200) remote: Counting objects: 73% (6716/9200) remote: Counting objects: 74% (6808/9200) remote: Counting objects: 75% (6900/9200) remote: Counting objects: 76% (6992/9200) remote: Counting objects: 77% (7084/9200) remote: Counting objects: 78% (7176/9200) remote: Counting objects: 79% (7268/9200) remote: Counting objects: 80% (7360/9200) remote: Counting objects: 81% (7452/9200) remote: Counting objects: 82% (7544/9200) remote: Counting objects: 83% (7636/9200) remote: Counting objects: 84% (7728/9200) remote: Counting objects: 85% (7820/9200) remote: Counting objects: 86% (7912/9200) remote: Counting objects: 87% (8004/9200) remote: Counting objects: 88% (8096/9200) remote: Counting objects: 89% (8188/9200) remote: Counting objects: 90% (8280/9200) remote: Counting objects: 91% (8372/9200) remote: Counting objects: 92% (8464/9200) remote: Counting objects: 93% (8556/9200) remote: Counting objects: 94% (8648/9200) remote: Counting objects: 95% (8740/9200) remote: Counting objects: 96% (8832/9200) remote: Counting objects: 97% (8924/9200) remote: Counting objects: 98% (9016/9200) remote: Counting objects: 99% (9108/9200) remote: Counting objects: 100% (9200/9200) remote: Counting objects: 100% (9200/9200), done. remote: Compressing objects: 0% (1/575) remote: Compressing objects: 1% (6/575) remote: Compressing objects: 2% (12/575) remote: Compressing objects: 3% (18/575) remote: Compressing objects: 4% (23/575) remote: Compressing objects: 5% (29/575) remote: Compressing objects: 6% (35/575) remote: Compressing objects: 7% (41/575) remote: Compressing objects: 8% (46/575) remote: Compressing objects: 9% (52/575) remote: Compressing objects: 10% (58/575) remote: Compressing objects: 11% (64/575) remote: Compressing objects: 12% (69/575) remote: Compressing objects: 13% (75/575) remote: Compressing objects: 14% (81/575) remote: Compressing objects: 15% (87/575) remote: Compressing objects: 16% (92/575) remote: Compressing objects: 17% (98/575) remote: Compressing objects: 18% (104/575) remote: Compressing objects: 19% (110/575) remote: Compressing objects: 20% (115/575) remote: Compressing objects: 21% (121/575) remote: Compressing objects: 22% (127/575) remote: Compressing objects: 23% (133/575) remote: Compressing objects: 24% (138/575) remote: Compressing objects: 25% (144/575) remote: Compressing objects: 26% (150/575) remote: Compressing objects: 27% (156/575) remote: Compressing objects: 28% (161/575) remote: Compressing objects: 29% (167/575) remote: Compressing objects: 30% (173/575) remote: Compressing objects: 31% (179/575) remote: Compressing objects: 32% (184/575) remote: Compressing objects: 33% (190/575) remote: Compressing objects: 34% (196/575) remote: Compressing objects: 35% (202/575) remote: Compressing objects: 36% (207/575) remote: Compressing objects: 37% (213/575) remote: Compressing objects: 38% (219/575) remote: Compressing objects: 39% (225/575) remote: Compressing objects: 40% (230/575) remote: Compressing objects: 41% (236/575) remote: Compressing objects: 42% (242/575) remote: Compressing objects: 43% (248/575) remote: Compressing objects: 44% (253/575) remote: Compressing objects: 45% (259/575) remote: Compressing objects: 46% (265/575) remote: Compressing objects: 47% (271/575) remote: Compressing objects: 48% (276/575) remote: Compressing objects: 49% (282/575) remote: Compressing objects: 50% (288/575) remote: Compressing objects: 51% (294/575) remote: Compressing objects: 52% (299/575) remote: Compressing objects: 53% (305/575) remote: Compressing objects: 54% (311/575) remote: Compressing objects: 55% (317/575) remote: Compressing objects: 56% (322/575) remote: Compressing objects: 57% (328/575) remote: Compressing objects: 58% (334/575) remote: Compressing objects: 59% (340/575) remote: Compressing objects: 60% (345/575) remote: Compressing objects: 61% (351/575) remote: Compressing objects: 62% (357/575) remote: Compressing objects: 63% (363/575) remote: Compressing objects: 64% (368/575) remote: Compressing objects: 65% (374/575) remote: Compressing objects: 66% (380/575) remote: Compressing objects: 67% (386/575) remote: Compressing objects: 68% (391/575) remote: Compressing objects: 69% (397/575) remote: Compressing objects: 70% (403/575) remote: Compressing objects: 71% (409/575) remote: Compressing objects: 72% (414/575) remote: Compressing objects: 73% (420/575) remote: Compressing objects: 74% (426/575) remote: Compressing objects: 75% (432/575) remote: Compressing objects: 76% (437/575) remote: Compressing objects: 77% (443/575) remote: Compressing objects: 78% (449/575) remote: Compressing objects: 79% (455/575) remote: Compressing objects: 80% (460/575) remote: Compressing objects: 81% (466/575) remote: Compressing objects: 82% (472/575) remote: Compressing objects: 83% (478/575) remote: Compressing objects: 84% (483/575) remote: Compressing objects: 85% (489/575) remote: Compressing objects: 86% (495/575) remote: Compressing objects: 87% (501/575) remote: Compressing objects: 88% (506/575) remote: Compressing objects: 89% (512/575) remote: Compressing objects: 90% (518/575) remote: Compressing objects: 91% (524/575) remote: Compressing objects: 92% (529/575) remote: Compressing objects: 93% (535/575) remote: Compressing objects: 94% (541/575) remote: Compressing objects: 95% (547/575) remote: Compressing objects: 96% (552/575) remote: Compressing objects: 97% (558/575) remote: Compressing objects: 98% (564/575) remote: Compressing objects: 99% (570/575) remote: Compressing objects: 100% (575/575) remote: Compressing objects: 100% (575/575), done. Receiving objects: 0% (1/16745) Receiving objects: 1% (168/16745) Receiving objects: 2% (335/16745) Receiving objects: 3% (503/16745) Receiving objects: 4% (670/16745) Receiving objects: 4% (799/16745), 1.68 MiB | 1.64 MiB/s Receiving objects: 5% (838/16745), 1.68 MiB | 1.64 MiB/s Receiving objects: 6% (1005/16745), 1.68 MiB | 1.64 MiB/s Receiving objects: 7% (1173/16745), 1.68 MiB | 1.64 MiB/s Receiving objects: 8% (1340/16745), 1.68 MiB | 1.64 MiB/s Receiving objects: 9% (1508/16745), 1.68 MiB | 1.64 MiB/s Receiving objects: 10% (1675/16745), 1.68 MiB | 1.64 MiB/s Receiving objects: 11% (1842/16745), 1.68 MiB | 1.64 MiB/s Receiving objects: 12% (2010/16745), 1.68 MiB | 1.64 MiB/s Receiving objects: 13% (2177/16745), 1.68 MiB | 1.64 MiB/s Receiving objects: 14% (2345/16745), 1.68 MiB | 1.64 MiB/s Receiving objects: 15% (2512/16745), 1.68 MiB | 1.64 MiB/s Receiving objects: 16% (2680/16745), 2.73 MiB | 1.79 MiB/s Receiving objects: 17% (2847/16745), 2.73 MiB | 1.79 MiB/s Receiving objects: 18% (3015/16745), 2.73 MiB | 1.79 MiB/s Receiving objects: 19% (3182/16745), 2.73 MiB | 1.79 MiB/s Receiving objects: 20% (3349/16745), 2.73 MiB | 1.79 MiB/s Receiving objects: 20% (3406/16745), 2.73 MiB | 1.79 MiB/s Receiving objects: 21% (3517/16745), 3.70 MiB | 1.83 MiB/s Receiving objects: 22% (3684/16745), 3.70 MiB | 1.83 MiB/s Receiving objects: 23% (3852/16745), 3.70 MiB | 1.83 MiB/s Receiving objects: 24% (4019/16745), 3.70 MiB | 1.83 MiB/s Receiving objects: 25% (4187/16745), 3.70 MiB | 1.83 MiB/s Receiving objects: 26% (4354/16745), 3.70 MiB | 1.83 MiB/s Receiving objects: 27% (4522/16745), 3.70 MiB | 1.83 MiB/s Receiving objects: 28% (4689/16745), 3.70 MiB | 1.83 MiB/s Receiving objects: 29% (4857/16745), 3.70 MiB | 1.83 MiB/s Receiving objects: 30% (5024/16745), 3.70 MiB | 1.83 MiB/s Receiving objects: 31% (5191/16745), 3.70 MiB | 1.83 MiB/s Receiving objects: 32% (5359/16745), 3.70 MiB | 1.83 MiB/s Receiving objects: 33% (5526/16745), 3.70 MiB | 1.83 MiB/s Receiving objects: 34% (5694/16745), 3.70 MiB | 1.83 MiB/s Receiving objects: 35% (5861/16745), 3.70 MiB | 1.83 MiB/s Receiving objects: 36% (6029/16745), 3.70 MiB | 1.83 MiB/s Receiving objects: 37% (6196/16745), 3.70 MiB | 1.83 MiB/s Receiving objects: 38% (6364/16745), 3.70 MiB | 1.83 MiB/s Receiving objects: 39% (6531/16745), 3.70 MiB | 1.83 MiB/s Receiving objects: 40% (6698/16745), 3.70 MiB | 1.83 MiB/s Receiving objects: 41% (6866/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 42% (7033/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 43% (7201/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 44% (7368/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 45% (7536/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 46% (7703/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 47% (7871/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 48% (8038/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 49% (8206/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 50% (8373/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 51% (8540/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 52% (8708/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 53% (8875/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 54% (9043/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 55% (9210/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 56% (9378/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 57% (9545/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 58% (9713/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 59% (9880/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 60% (10047/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 61% (10215/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 62% (10382/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 63% (10550/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 64% (10717/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 65% (10885/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 66% (11052/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 67% (11220/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 68% (11387/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 69% (11555/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 70% (11722/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 71% (11889/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 72% (12057/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 73% (12224/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 74% (12392/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 75% (12559/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 76% (12727/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 77% (12894/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 78% (13062/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 79% (13229/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 80% (13396/16745), 5.03 MiB | 1.99 MiB/s Receiving objects: 80% (13425/16745), 6.13 MiB | 1.97 MiB/s Receiving objects: 81% (13564/16745), 6.13 MiB | 1.97 MiB/s Receiving objects: 82% (13731/16745), 6.13 MiB | 1.97 MiB/s Receiving objects: 83% (13899/16745), 6.13 MiB | 1.97 MiB/s Receiving objects: 84% (14066/16745), 6.13 MiB | 1.97 MiB/s Receiving objects: 85% (14234/16745), 6.13 MiB | 1.97 MiB/s Receiving objects: 86% (14401/16745), 6.13 MiB | 1.97 MiB/s Receiving objects: 87% (14569/16745), 6.13 MiB | 1.97 MiB/s Receiving objects: 88% (14736/16745), 6.13 MiB | 1.97 MiB/s Receiving objects: 89% (14904/16745), 6.13 MiB | 1.97 MiB/s Receiving objects: 90% (15071/16745), 7.82 MiB | 2.17 MiB/s Receiving objects: 91% (15238/16745), 7.82 MiB | 2.17 MiB/s Receiving objects: 92% (15406/16745), 7.82 MiB | 2.17 MiB/s Receiving objects: 93% (15573/16745), 7.82 MiB | 2.17 MiB/s Receiving objects: 93% (15705/16745), 7.82 MiB | 2.17 MiB/s Receiving objects: 94% (15741/16745), 7.82 MiB | 2.17 MiB/s Receiving objects: 95% (15908/16745), 9.05 MiB | 2.19 MiB/s Receiving objects: 96% (16076/16745), 9.05 MiB | 2.19 MiB/s Receiving objects: 97% (16243/16745), 9.05 MiB | 2.19 MiB/s Receiving objects: 98% (16411/16745), 9.05 MiB | 2.19 MiB/s Receiving objects: 99% (16578/16745), 9.05 MiB | 2.19 MiB/s remote: Total 16745 (delta 8716), reused 8981 (delta 8544), pack-reused 7545 Receiving objects: 100% (16745/16745), 9.05 MiB | 2.19 MiB/s Receiving objects: 100% (16745/16745), 10.19 MiB | 2.23 MiB/s, done. Resolving deltas: 0% (0/12118) Resolving deltas: 1% (122/12118) Resolving deltas: 2% (243/12118) Resolving deltas: 3% (364/12118) Resolving deltas: 4% (485/12118) Resolving deltas: 5% (606/12118) Resolving deltas: 6% (728/12118) Resolving deltas: 7% (849/12118) Resolving deltas: 8% (970/12118) Resolving deltas: 9% (1091/12118) Resolving deltas: 10% (1212/12118) Resolving deltas: 11% (1333/12118) Resolving deltas: 12% (1455/12118) Resolving deltas: 13% (1576/12118) Resolving deltas: 14% (1697/12118) Resolving deltas: 15% (1818/12118) Resolving deltas: 16% (1939/12118) Resolving deltas: 17% (2061/12118) Resolving deltas: 18% (2183/12118) Resolving deltas: 19% (2303/12118) Resolving deltas: 20% (2424/12118) Resolving deltas: 21% (2545/12118) Resolving deltas: 22% (2666/12118) Resolving deltas: 23% (2788/12118) Resolving deltas: 24% (2909/12118) Resolving deltas: 25% (3030/12118) Resolving deltas: 26% (3151/12118) Resolving deltas: 27% (3272/12118) Resolving deltas: 28% (3394/12118) Resolving deltas: 29% (3515/12118) Resolving deltas: 30% (3636/12118) Resolving deltas: 31% (3757/12118) Resolving deltas: 32% (3878/12118) Resolving deltas: 33% (3999/12118) Resolving deltas: 34% (4121/12118) Resolving deltas: 35% (4242/12118) Resolving deltas: 36% (4363/12118) Resolving deltas: 37% (4484/12118) Resolving deltas: 38% (4605/12118) Resolving deltas: 39% (4727/12118) Resolving deltas: 40% (4848/12118) Resolving deltas: 41% (4969/12118) Resolving deltas: 42% (5090/12118) Resolving deltas: 43% (5211/12118) Resolving deltas: 44% (5332/12118) Resolving deltas: 45% (5454/12118) Resolving deltas: 46% (5575/12118) Resolving deltas: 47% (5696/12118) Resolving deltas: 48% (5817/12118) Resolving deltas: 49% (5938/12118) Resolving deltas: 50% (6059/12118) Resolving deltas: 51% (6181/12118) Resolving deltas: 52% (6302/12118) Resolving deltas: 53% (6423/12118) Resolving deltas: 54% (6544/12118) Resolving deltas: 55% (6665/12118) Resolving deltas: 56% (6787/12118) Resolving deltas: 57% (6908/12118) Resolving deltas: 58% (7029/12118) Resolving deltas: 59% (7150/12118) Resolving deltas: 60% (7271/12118) Resolving deltas: 61% (7392/12118) Resolving deltas: 62% (7514/12118) Resolving deltas: 63% (7635/12118) Resolving deltas: 64% (7756/12118) Resolving deltas: 65% (7877/12118) Resolving deltas: 66% (7998/12118) Resolving deltas: 67% (8120/12118) Resolving deltas: 68% (8241/12118) Resolving deltas: 69% (8362/12118) Resolving deltas: 70% (8483/12118) Resolving deltas: 70% (8553/12118) Resolving deltas: 71% (8604/12118) Resolving deltas: 72% (8725/12118) Resolving deltas: 73% (8847/12118) Resolving deltas: 74% (8968/12118) Resolving deltas: 74% (8974/12118) Resolving deltas: 75% (9089/12118) Resolving deltas: 76% (9210/12118) Resolving deltas: 77% (9331/12118) Resolving deltas: 78% (9453/12118) Resolving deltas: 78% (9550/12118) Resolving deltas: 79% (9574/12118) Resolving deltas: 80% (9695/12118) Resolving deltas: 81% (9816/12118) Resolving deltas: 81% (9887/12118) Resolving deltas: 82% (9937/12118) Resolving deltas: 83% (10058/12118) Resolving deltas: 84% (10180/12118) Resolving deltas: 85% (10301/12118) Resolving deltas: 86% (10422/12118) Resolving deltas: 86% (10475/12118) Resolving deltas: 87% (10543/12118) Resolving deltas: 88% (10664/12118) Resolving deltas: 89% (10786/12118) Resolving deltas: 89% (10846/12118) Resolving deltas: 90% (10907/12118) Resolving deltas: 90% (10932/12118) Resolving deltas: 91% (11028/12118) Resolving deltas: 91% (11115/12118) Resolving deltas: 92% (11149/12118) Resolving deltas: 93% (11270/12118) Resolving deltas: 94% (11391/12118) Resolving deltas: 95% (11513/12118) Resolving deltas: 96% (11634/12118) Resolving deltas: 96% (11643/12118) Resolving deltas: 97% (11755/12118) Resolving deltas: 98% (11876/12118) Resolving deltas: 98% (11912/12118) Resolving deltas: 99% (11997/12118) Resolving deltas: 100% (12118/12118) Resolving deltas: 100% (12118/12118), completed with 1188 local objects. * [new branch] 4154-docs-remove-cta -> 4154-docs-remove-cta * [new branch] 56quarters/async-cache-writes -> 56quarters/async-cache-writes * [new branch] 56quarters/enable-redis-support -> 56quarters/enable-redis-support * [new branch] 56quarters/memcached-sizes -> 56quarters/memcached-sizes * [new branch] add-compactor-grace-period -> add-compactor-grace-period * [new branch] add-overrides-extensions -> add-overrides-extensions * [new branch] add-pooling-to-chunks-fetched-from-cache -> add-pooling-to-chunks-fetched-from-cache * [new branch] aggregations-v3 -> aggregations-v3 * [new branch] aldernero/release-notes-2.7 -> aldernero/release-notes-2.7 * [new branch] arve/goupdate -> arve/goupdate * [new branch] charleskorn/content-negotation-improvements -> charleskorn/content-negotation-improvements * [new branch] charleskorn/new-query-result-format-in-ruler -> charleskorn/new-query-result-format-in-ruler * [new branch] charleskorn/new-query-result-format-on-main -> charleskorn/new-query-result-format-on-main * [new branch] charleskorn/new-query-result-format-query-frontend -> charleskorn/new-query-result-format-query-frontend 20e5d0044..f18727fa6 charleskorn/rule-evaluation-tool -> charleskorn/rule-evaluation-tool * [new branch] charleskorn/thanos-promql-engine-poc -> charleskorn/thanos-promql-engine-poc * [new branch] chore/improve-comments -> chore/improve-comments * [new branch] codesome/sync-frontend -> codesome/sync-frontend * [new branch] codesome/sync-querier -> codesome/sync-querier * [new branch] compactor.consistency-delay -> compactor.consistency-delay * [new branch] debugging_runners -> debugging_runners * [new branch] deprecate-max-series-per-metric -> deprecate-max-series-per-metric * [new branch] dimitar/chunks-sizes-analysis -> dimitar/chunks-sizes-analysis * [new branch] dimitar/r222-with-partitioner-metrics -> dimitar/r222-with-partitioner-metrics * [new branch] dimitar/r222-with-reduced-batches -> dimitar/r222-with-reduced-batches * [new branch] dimitar/r223-with-chunks-and-series-counting -> dimitar/r223-with-chunks-and-series-counting * [new branch] dimitar/store-gateway-chunk-range-reading -> dimitar/store-gateway-chunk-range-reading + 8b232973b...67077ab43 dimitar/store-gateway-chunks-grouping -> dimitar/store-gateway-chunks-grouping (forced update) * [new branch] dimitar/store-gateway-chunks-grouping-v2 -> dimitar/store-gateway-chunks-grouping-v2 * [new branch] dimitar/store-gateway-chunks-grouping-v3-noop-cache -> dimitar/store-gateway-chunks-grouping-v3-noop-cache * [new branch] dimitar/store-gateway-chunks-metrics -> dimitar/store-gateway-chunks-metrics * [new branch] dimitar/streaming-label-names -> dimitar/streaming-label-names * [new branch] extend_forwarding_limits_struct -> extend_forwarding_limits_struct * [new branch] felix/almost-up-to-date -> felix/almost-up-to-date * [new branch] fl/ingesters-within-per-tenant -> fl/ingesters-within-per-tenant * [new branch] format-comment -> format-comment * [new branch] gem-release-2.6 -> gem-release-2.6 * [new branch] helm-chart-weekly-4.1.0-weekly.221 -> helm-chart-weekly-4.1.0-weekly.221 * [new branch] helm-chart-weekly-4.1.0-weekly.226 -> helm-chart-weekly-4.1.0-weekly.226 * [new branch] investigate-store-gateway-slow-send -> investigate-store-gateway-slow-send * [new branch] joeblubaugh/mimirtool -> joeblubaugh/mimirtool * [new branch] julienduchesne/set-output-is-deprecated -> julienduchesne/set-output-is-deprecated * [new branch] krajo/20230125-default-json-decoder -> krajo/20230125-default-json-decoder * [new branch] krajo/distributor-no-validate -> krajo/distributor-no-validate * [new branch] krajo/fix-ingester-appender-usage -> krajo/fix-ingester-appender-usage * [new branch] krajo/from-radudd -> krajo/from-radudd * [new branch] krajo/improve-helm-release-md -> krajo/improve-helm-release-md * [new branch] krajo/mockingester -> krajo/mockingester + 7beb731d8...07c1c8a89 labels-as-string -> labels-as-string (forced update) * [new branch] lamida/helm-redis -> lamida/helm-redis * [new branch] lamida/jsonnet-redis -> lamida/jsonnet-redis * [new branch] leizor/compactor-integration-test -> leizor/compactor-integration-test * [new branch] leizor/mimir/issues/3478 -> leizor/mimir/issues/3478 * [new branch] leizor/mimir/issues/3715 -> leizor/mimir/issues/3715 + 3b4479d87...40905bc2c leizor/query-squad/issues/263 -> leizor/query-squad/issues/263 (forced update) * [new branch] leizor/science -> leizor/science 0a5e0e33a..eae106570 main -> main * [new branch] make-results-cache-ttl-configurable -> make-results-cache-ttl-configurable * [new branch] mimir-distributed-release-4.1 -> mimir-distributed-release-4.1 * [new branch] mimir-distributed-release-4.2 -> mimir-distributed-release-4.2 + c11ffef48...65aae5620 mixin-include-container-logs-link -> mixin-include-container-logs-link (forced update) * [new branch] njpm/improving-ooo-test -> njpm/improving-ooo-test * [new branch] osg-grafana/2023-02-24-move-ha-deduplication-consul-to-helm-chart -> osg-grafana/2023-02-24-move-ha-deduplication-consul-to-helm-chart * [new branch] otlp-dep-update -> otlp-dep-update * [new branch] query-stats-loki-rules -> query-stats-loki-rules * [new branch] r222 -> r222 * [new branch] r223 -> r223 * [new branch] r224 -> r224 * [new branch] r225 -> r225 * [new branch] r226 -> r226 * [new branch] r227 -> r227 * [new branch] r228 -> r228 * [new branch] read-write-deployment/docs2 -> read-write-deployment/docs2 * [new branch] register-extensions -> register-extensions c586d4c21..77a2da5c6 release-2.2 -> release-2.2 75576aa75..e761afe5f release-2.3 -> release-2.3 c2eaf3b3a..c53b039df release-2.4 -> release-2.4 1852e09b0..b22d121d5 release-2.5 -> release-2.5 * [new branch] release-2.6 -> release-2.6 * [new branch] remove-non-streaming -> remove-non-streaming 9a2e392f1..986e84de5 sparsehistogram -> sparsehistogram + f721c7b8f...67fa1f2d8 sparsehistogram-goes-boom -> sparsehistogram-goes-boom (forced update) + bcafdd171...2445dad84 sparsehistogram-test -> sparsehistogram-test (forced update) + cbb10d89f...51bfd78c3 store-gateway-load-test -> store-gateway-load-test (forced update) * [new branch] the-most-stupid-chunks-cache-implementation -> the-most-stupid-chunks-cache-implementation * [new branch] updatekube -> updatekube * [new branch] updateopenapi -> updateopenapi * [new branch] yuri-tceretian/alertmanager-grafana -> yuri-tceretian/alertmanager-grafana * [new branch] yuri-test-branch -> yuri-test-branch * [new branch] yuri/integration-test-querier-sg-422 -> yuri/integration-test-querier-sg-422 * [new branch] yuri/wal-replay-optimization -> yuri/wal-replay-optimization + 9e21e1b9d...673f09e11 refs/pull/3295/merge -> refs/pull/3295/merge (forced update) + 2ae93f55f...1389a33d2 refs/pull/3411/head -> refs/pull/3411/head (forced update) 9a2e392f1..986e84de5 refs/pull/3505/head -> refs/pull/3505/head + aeb91ca69...425d55dbf refs/pull/3505/merge -> refs/pull/3505/merge (forced update) + 336e165fb...6dea5c878 refs/pull/3541/head -> refs/pull/3541/head (forced update) 9781bde19..223410c28 refs/pull/3554/head -> refs/pull/3554/head + 7beb731d8...07c1c8a89 refs/pull/3555/head -> refs/pull/3555/head (forced update) + c9056ae04...65ec07f36 refs/pull/3555/merge -> refs/pull/3555/merge (forced update) + 435a56ebd...6c0f640ae refs/pull/3619/merge -> refs/pull/3619/merge (forced update) + 4ea6b91a1...f38c7a337 refs/pull/3629/head -> refs/pull/3629/head (forced update) + f721c7b8f...67fa1f2d8 refs/pull/3707/head -> refs/pull/3707/head (forced update) + c38cd708b...0ad3e81f2 refs/pull/3707/merge -> refs/pull/3707/merge (forced update) + ff47cf608...be3e65e54 refs/pull/3740/head -> refs/pull/3740/head (forced update) + 90e8ed4f7...6b77d3726 refs/pull/3852/head -> refs/pull/3852/head (forced update) + bcafdd171...2445dad84 refs/pull/3858/head -> refs/pull/3858/head (forced update) + 140dbb16b...03e7faea3 refs/pull/3858/merge -> refs/pull/3858/merge (forced update) ae755b895..a6e21a5ef refs/pull/3872/head -> refs/pull/3872/head + 941ac0ab6...c9093f05b refs/pull/3911/head -> refs/pull/3911/head (forced update) + 89810ab6b...db4c75855 refs/pull/3930/head -> refs/pull/3930/head (forced update) 6b8a3383c..834886a2c refs/pull/3948/head -> refs/pull/3948/head 173e1b194..abce6974a refs/pull/3950/head -> refs/pull/3950/head 251b40bdd..c8e69647c refs/pull/3951/head -> refs/pull/3951/head f4f41129c..4d90d3a8f refs/pull/3952/head -> refs/pull/3952/head 8b55ae57c..8ebdb35ec refs/pull/3956/head -> refs/pull/3956/head + 294bf073a...1a751a02f refs/pull/3957/merge -> refs/pull/3957/merge (forced update) 06fae2bdd..734ccb49d refs/pull/3962/head -> refs/pull/3962/head + c11ffef48...65aae5620 refs/pull/3964/head -> refs/pull/3964/head (forced update) + c96cc1f21...c17331c81 refs/pull/3964/merge -> refs/pull/3964/merge (forced update) + 8b232973b...67077ab43 refs/pull/3968/head -> refs/pull/3968/head (forced update) 2a020de53..a9a8d08f2 refs/pull/3978/head -> refs/pull/3978/head * [new ref] refs/pull/3983/head -> refs/pull/3983/head * [new ref] refs/pull/3984/head -> refs/pull/3984/head * [new ref] refs/pull/3985/head -> refs/pull/3985/head * [new ref] refs/pull/3989/head -> refs/pull/3989/head * [new ref] refs/pull/3991/head -> refs/pull/3991/head * [new ref] refs/pull/3993/head -> refs/pull/3993/head * [new ref] refs/pull/3995/head -> refs/pull/3995/head * [new ref] refs/pull/3996/head -> refs/pull/3996/head * [new ref] refs/pull/3997/head -> refs/pull/3997/head * [new ref] refs/pull/3998/head -> refs/pull/3998/head * [new ref] refs/pull/3999/head -> refs/pull/3999/head * [new ref] refs/pull/4001/head -> refs/pull/4001/head * [new ref] refs/pull/4002/head -> refs/pull/4002/head * [new ref] refs/pull/4003/head -> refs/pull/4003/head * [new ref] refs/pull/4004/head -> refs/pull/4004/head * [new ref] refs/pull/4006/head -> refs/pull/4006/head * [new ref] refs/pull/4007/head -> refs/pull/4007/head * [new ref] refs/pull/4008/head -> refs/pull/4008/head * [new ref] refs/pull/4011/head -> refs/pull/4011/head * [new ref] refs/pull/4012/head -> refs/pull/4012/head * [new ref] refs/pull/4013/head -> refs/pull/4013/head * [new ref] refs/pull/4014/head -> refs/pull/4014/head * [new ref] refs/pull/4015/head -> refs/pull/4015/head * [new ref] refs/pull/4016/head -> refs/pull/4016/head * [new ref] refs/pull/4017/head -> refs/pull/4017/head * [new ref] refs/pull/4018/head -> refs/pull/4018/head * [new ref] refs/pull/4019/head -> refs/pull/4019/head * [new ref] refs/pull/4020/head -> refs/pull/4020/head * [new ref] refs/pull/4022/head -> refs/pull/4022/head * [new ref] refs/pull/4023/head -> refs/pull/4023/head * [new ref] refs/pull/4023/merge -> refs/pull/4023/merge * [new ref] refs/pull/4024/head -> refs/pull/4024/head * [new ref] refs/pull/4024/merge -> refs/pull/4024/merge * [new ref] refs/pull/4025/head -> refs/pull/4025/head * [new ref] refs/pull/4026/head -> refs/pull/4026/head * [new ref] refs/pull/4027/head -> refs/pull/4027/head * [new ref] refs/pull/4028/head -> refs/pull/4028/head * [new ref] refs/pull/4029/head -> refs/pull/4029/head * [new ref] refs/pull/4030/head -> refs/pull/4030/head * [new ref] refs/pull/4031/head -> refs/pull/4031/head * [new ref] refs/pull/4032/head -> refs/pull/4032/head * [new ref] refs/pull/4033/head -> refs/pull/4033/head * [new ref] refs/pull/4034/head -> refs/pull/4034/head * [new ref] refs/pull/4035/head -> refs/pull/4035/head * [new ref] refs/pull/4036/head -> refs/pull/4036/head * [new ref] refs/pull/4037/head -> refs/pull/4037/head * [new ref] refs/pull/4038/head -> refs/pull/4038/head * [new ref] refs/pull/4039/head -> refs/pull/4039/head * [new ref] refs/pull/4041/head -> refs/pull/4041/head * [new ref] refs/pull/4042/head -> refs/pull/4042/head * [new ref] refs/pull/4043/head -> refs/pull/4043/head * [new ref] refs/pull/4045/head -> refs/pull/4045/head * [new ref] refs/pull/4046/head -> refs/pull/4046/head * [new ref] refs/pull/4049/head -> refs/pull/4049/head * [new ref] refs/pull/4051/head -> refs/pull/4051/head * [new ref] refs/pull/4052/head -> refs/pull/4052/head * [new ref] refs/pull/4054/head -> refs/pull/4054/head * [new ref] refs/pull/4055/head -> refs/pull/4055/head * [new ref] refs/pull/4056/head -> refs/pull/4056/head * [new ref] refs/pull/4057/head -> refs/pull/4057/head * [new ref] refs/pull/4058/head -> refs/pull/4058/head * [new ref] refs/pull/4059/head -> refs/pull/4059/head * [new ref] refs/pull/4059/merge -> refs/pull/4059/merge * [new ref] refs/pull/4060/head -> refs/pull/4060/head * [new ref] refs/pull/4061/head -> refs/pull/4061/head * [new ref] refs/pull/4062/head -> refs/pull/4062/head * [new ref] refs/pull/4063/head -> refs/pull/4063/head * [new ref] refs/pull/4063/merge -> refs/pull/4063/merge * [new ref] refs/pull/4065/head -> refs/pull/4065/head * [new ref] refs/pull/4067/head -> refs/pull/4067/head * [new ref] refs/pull/4072/head -> refs/pull/4072/head * [new ref] refs/pull/4073/head -> refs/pull/4073/head * [new ref] refs/pull/4074/head -> refs/pull/4074/head * [new ref] refs/pull/4075/head -> refs/pull/4075/head * [new ref] refs/pull/4076/head -> refs/pull/4076/head * [new ref] refs/pull/4078/head -> refs/pull/4078/head * [new ref] refs/pull/4079/head -> refs/pull/4079/head * [new ref] refs/pull/4080/head -> refs/pull/4080/head * [new ref] refs/pull/4081/head -> refs/pull/4081/head * [new ref] refs/pull/4083/head -> refs/pull/4083/head * [new ref] refs/pull/4084/head -> refs/pull/4084/head * [new ref] refs/pull/4084/merge -> refs/pull/4084/merge * [new ref] refs/pull/4087/head -> refs/pull/4087/head * [new ref] refs/pull/4088/head -> refs/pull/4088/head * [new ref] refs/pull/4089/head -> refs/pull/4089/head * [new ref] refs/pull/4090/head -> refs/pull/4090/head * [new ref] refs/pull/4091/head -> refs/pull/4091/head * [new ref] refs/pull/4093/head -> refs/pull/4093/head * [new ref] refs/pull/4095/head -> refs/pull/4095/head * [new ref] refs/pull/4096/head -> refs/pull/4096/head * [new ref] refs/pull/4097/head -> refs/pull/4097/head * [new ref] refs/pull/4097/merge -> refs/pull/4097/merge * [new ref] refs/pull/4099/head -> refs/pull/4099/head * [new ref] refs/pull/4100/head -> refs/pull/4100/head * [new ref] refs/pull/4101/head -> refs/pull/4101/head * [new ref] refs/pull/4102/head -> refs/pull/4102/head * [new ref] refs/pull/4103/head -> refs/pull/4103/head * [new ref] refs/pull/4106/head -> refs/pull/4106/head * [new ref] refs/pull/4107/head -> refs/pull/4107/head * [new ref] refs/pull/4110/head -> refs/pull/4110/head * [new ref] refs/pull/4111/head -> refs/pull/4111/head * [new ref] refs/pull/4112/head -> refs/pull/4112/head * [new ref] refs/pull/4113/head -> refs/pull/4113/head * [new ref] refs/pull/4114/head -> refs/pull/4114/head * [new ref] refs/pull/4116/head -> refs/pull/4116/head * [new ref] refs/pull/4119/head -> refs/pull/4119/head * [new ref] refs/pull/4120/head -> refs/pull/4120/head * [new ref] refs/pull/4121/head -> refs/pull/4121/head * [new ref] refs/pull/4122/head -> refs/pull/4122/head * [new ref] refs/pull/4123/head -> refs/pull/4123/head * [new ref] refs/pull/4123/merge -> refs/pull/4123/merge * [new ref] refs/pull/4124/head -> refs/pull/4124/head * [new ref] refs/pull/4125/head -> refs/pull/4125/head * [new ref] refs/pull/4126/head -> refs/pull/4126/head * [new ref] refs/pull/4126/merge -> refs/pull/4126/merge * [new ref] refs/pull/4127/head -> refs/pull/4127/head * [new ref] refs/pull/4128/head -> refs/pull/4128/head * [new ref] refs/pull/4129/head -> refs/pull/4129/head * [new ref] refs/pull/4130/head -> refs/pull/4130/head * [new ref] refs/pull/4131/head -> refs/pull/4131/head * [new ref] refs/pull/4132/head -> refs/pull/4132/head * [new ref] refs/pull/4133/head -> refs/pull/4133/head * [new ref] refs/pull/4135/head -> refs/pull/4135/head * [new ref] refs/pull/4136/head -> refs/pull/4136/head * [new ref] refs/pull/4140/head -> refs/pull/4140/head * [new ref] refs/pull/4141/head -> refs/pull/4141/head * [new ref] refs/pull/4142/head -> refs/pull/4142/head * [new ref] refs/pull/4143/head -> refs/pull/4143/head * [new ref] refs/pull/4145/head -> refs/pull/4145/head * [new ref] refs/pull/4146/head -> refs/pull/4146/head * [new ref] refs/pull/4147/head -> refs/pull/4147/head * [new ref] refs/pull/4148/head -> refs/pull/4148/head * [new ref] refs/pull/4149/head -> refs/pull/4149/head * [new ref] refs/pull/4150/head -> refs/pull/4150/head * [new ref] refs/pull/4151/head -> refs/pull/4151/head * [new ref] refs/pull/4152/head -> refs/pull/4152/head * [new ref] refs/pull/4153/head -> refs/pull/4153/head * [new ref] refs/pull/4156/head -> refs/pull/4156/head * [new ref] refs/pull/4159/head -> refs/pull/4159/head * [new ref] refs/pull/4161/head -> refs/pull/4161/head * [new ref] refs/pull/4161/merge -> refs/pull/4161/merge * [new ref] refs/pull/4162/head -> refs/pull/4162/head * [new ref] refs/pull/4163/head -> refs/pull/4163/head * [new ref] refs/pull/4164/head -> refs/pull/4164/head * [new ref] refs/pull/4165/head -> refs/pull/4165/head * [new ref] refs/pull/4166/head -> refs/pull/4166/head * [new ref] refs/pull/4168/head -> refs/pull/4168/head * [new ref] refs/pull/4169/head -> refs/pull/4169/head * [new ref] refs/pull/4170/head -> refs/pull/4170/head * [new ref] refs/pull/4171/head -> refs/pull/4171/head * [new ref] refs/pull/4172/head -> refs/pull/4172/head * [new ref] refs/pull/4174/head -> refs/pull/4174/head * [new ref] refs/pull/4177/head -> refs/pull/4177/head * [new ref] refs/pull/4178/head -> refs/pull/4178/head * [new ref] refs/pull/4179/head -> refs/pull/4179/head * [new ref] refs/pull/4180/head -> refs/pull/4180/head * [new ref] refs/pull/4182/head -> refs/pull/4182/head * [new ref] refs/pull/4183/head -> refs/pull/4183/head * [new ref] refs/pull/4183/merge -> refs/pull/4183/merge * [new ref] refs/pull/4186/head -> refs/pull/4186/head * [new ref] refs/pull/4188/head -> refs/pull/4188/head * [new ref] refs/pull/4190/head -> refs/pull/4190/head * [new ref] refs/pull/4193/head -> refs/pull/4193/head * [new ref] refs/pull/4194/head -> refs/pull/4194/head * [new ref] refs/pull/4195/head -> refs/pull/4195/head * [new ref] refs/pull/4196/head -> refs/pull/4196/head * [new ref] refs/pull/4197/head -> refs/pull/4197/head * [new ref] refs/pull/4197/merge -> refs/pull/4197/merge * [new ref] refs/pull/4199/head -> refs/pull/4199/head * [new ref] refs/pull/4201/head -> refs/pull/4201/head * [new ref] refs/pull/4202/head -> refs/pull/4202/head * [new ref] refs/pull/4203/head -> refs/pull/4203/head * [new ref] refs/pull/4205/head -> refs/pull/4205/head * [new ref] refs/pull/4206/head -> refs/pull/4206/head * [new ref] refs/pull/4207/head -> refs/pull/4207/head * [new ref] refs/pull/4207/merge -> refs/pull/4207/merge * [new ref] refs/pull/4209/head -> refs/pull/4209/head * [new ref] refs/pull/4210/head -> refs/pull/4210/head * [new ref] refs/pull/4211/head -> refs/pull/4211/head * [new ref] refs/pull/4213/head -> refs/pull/4213/head * [new ref] refs/pull/4216/head -> refs/pull/4216/head * [new ref] refs/pull/4217/head -> refs/pull/4217/head * [new ref] refs/pull/4218/head -> refs/pull/4218/head * [new ref] refs/pull/4219/head -> refs/pull/4219/head * [new ref] refs/pull/4220/head -> refs/pull/4220/head * [new ref] refs/pull/4221/head -> refs/pull/4221/head * [new ref] refs/pull/4223/head -> refs/pull/4223/head * [new ref] refs/pull/4225/head -> refs/pull/4225/head * [new ref] refs/pull/4227/head -> refs/pull/4227/head * [new ref] refs/pull/4229/head -> refs/pull/4229/head * [new ref] refs/pull/4229/merge -> refs/pull/4229/merge * [new ref] refs/pull/4230/head -> refs/pull/4230/head * [new ref] refs/pull/4231/head -> refs/pull/4231/head * [new ref] refs/pull/4233/head -> refs/pull/4233/head * [new ref] refs/pull/4234/head -> refs/pull/4234/head * [new ref] refs/pull/4234/merge -> refs/pull/4234/merge * [new ref] refs/pull/4237/head -> refs/pull/4237/head * [new ref] refs/pull/4238/head -> refs/pull/4238/head * [new ref] refs/pull/4240/head -> refs/pull/4240/head * [new ref] refs/pull/4242/head -> refs/pull/4242/head * [new ref] refs/pull/4244/head -> refs/pull/4244/head * [new ref] refs/pull/4246/head -> refs/pull/4246/head * [new ref] refs/pull/4247/head -> refs/pull/4247/head * [new ref] refs/pull/4248/head -> refs/pull/4248/head * [new ref] refs/pull/4249/head -> refs/pull/4249/head * [new ref] refs/pull/4250/head -> refs/pull/4250/head * [new ref] refs/pull/4251/head -> refs/pull/4251/head * [new ref] refs/pull/4252/head -> refs/pull/4252/head * [new ref] refs/pull/4253/head -> refs/pull/4253/head * [new ref] refs/pull/4254/head -> refs/pull/4254/head * [new ref] refs/pull/4255/head -> refs/pull/4255/head * [new ref] refs/pull/4255/merge -> refs/pull/4255/merge * [new ref] refs/pull/4256/head -> refs/pull/4256/head * [new ref] refs/pull/4257/head -> refs/pull/4257/head * [new ref] refs/pull/4258/head -> refs/pull/4258/head * [new ref] refs/pull/4262/head -> refs/pull/4262/head * [new ref] refs/pull/4263/head -> refs/pull/4263/head * [new ref] refs/pull/4264/head -> refs/pull/4264/head * [new ref] refs/pull/4265/head -> refs/pull/4265/head * [new ref] refs/pull/4266/head -> refs/pull/4266/head * [new ref] refs/pull/4268/head -> refs/pull/4268/head * [new ref] refs/pull/4269/head -> refs/pull/4269/head * [new ref] refs/pull/4270/head -> refs/pull/4270/head * [new ref] refs/pull/4271/head -> refs/pull/4271/head * [new ref] refs/pull/4272/head -> refs/pull/4272/head * [new ref] refs/pull/4272/merge -> refs/pull/4272/merge * [new ref] refs/pull/4273/head -> refs/pull/4273/head * [new ref] refs/pull/4276/head -> refs/pull/4276/head * [new ref] refs/pull/4278/head -> refs/pull/4278/head * [new ref] refs/pull/4279/head -> refs/pull/4279/head * [new ref] refs/pull/4280/head -> refs/pull/4280/head * [new ref] refs/pull/4281/head -> refs/pull/4281/head * [new ref] refs/pull/4282/head -> refs/pull/4282/head * [new ref] refs/pull/4283/head -> refs/pull/4283/head * [new ref] refs/pull/4285/head -> refs/pull/4285/head * [new ref] refs/pull/4286/head -> refs/pull/4286/head * [new ref] refs/pull/4287/head -> refs/pull/4287/head * [new ref] refs/pull/4287/merge -> refs/pull/4287/merge * [new ref] refs/pull/4289/head -> refs/pull/4289/head * [new ref] refs/pull/4290/head -> refs/pull/4290/head * [new ref] refs/pull/4294/head -> refs/pull/4294/head * [new ref] refs/pull/4295/head -> refs/pull/4295/head * [new ref] refs/pull/4295/merge -> refs/pull/4295/merge * [new ref] refs/pull/4296/head -> refs/pull/4296/head * [new ref] refs/pull/4297/head -> refs/pull/4297/head * [new ref] refs/pull/4298/head -> refs/pull/4298/head * [new ref] refs/pull/4299/head -> refs/pull/4299/head * [new ref] refs/pull/4300/head -> refs/pull/4300/head * [new ref] refs/pull/4300/merge -> refs/pull/4300/merge * [new ref] refs/pull/4301/head -> refs/pull/4301/head * [new ref] refs/pull/4301/merge -> refs/pull/4301/merge * [new ref] refs/pull/4302/head -> refs/pull/4302/head * [new ref] refs/pull/4302/merge -> refs/pull/4302/merge * [new ref] refs/pull/4303/head -> refs/pull/4303/head * [new ref] refs/pull/4304/head -> refs/pull/4304/head * [new ref] refs/pull/4305/head -> refs/pull/4305/head * [new ref] refs/pull/4306/head -> refs/pull/4306/head * [new ref] refs/pull/4307/head -> refs/pull/4307/head * [new ref] refs/pull/4308/head -> refs/pull/4308/head * [new ref] refs/pull/4309/head -> refs/pull/4309/head * [new ref] refs/pull/4312/head -> refs/pull/4312/head * [new ref] refs/pull/4314/head -> refs/pull/4314/head * [new ref] refs/pull/4314/merge -> refs/pull/4314/merge * [new ref] refs/pull/4315/head -> refs/pull/4315/head * [new ref] refs/pull/4316/head -> refs/pull/4316/head * [new ref] refs/pull/4317/head -> refs/pull/4317/head * [new ref] refs/pull/4318/head -> refs/pull/4318/head * [new ref] refs/pull/4319/head -> refs/pull/4319/head * [new ref] refs/pull/4322/head -> refs/pull/4322/head * [new ref] refs/pull/4323/head -> refs/pull/4323/head * [new ref] refs/pull/4324/head -> refs/pull/4324/head * [new ref] refs/pull/4326/head -> refs/pull/4326/head * [new ref] refs/pull/4328/head -> refs/pull/4328/head * [new ref] refs/pull/4329/head -> refs/pull/4329/head * [new ref] refs/pull/4330/head -> refs/pull/4330/head * [new ref] refs/pull/4331/head -> refs/pull/4331/head * [new ref] refs/pull/4331/merge -> refs/pull/4331/merge * [new ref] refs/pull/4332/head -> refs/pull/4332/head * [new ref] refs/pull/4332/merge -> refs/pull/4332/merge * [new ref] refs/pull/4334/head -> refs/pull/4334/head * [new ref] refs/pull/4337/head -> refs/pull/4337/head * [new ref] refs/pull/4338/head -> refs/pull/4338/head * [new ref] refs/pull/4339/head -> refs/pull/4339/head * [new ref] refs/pull/4340/head -> refs/pull/4340/head * [new ref] refs/pull/4342/head -> refs/pull/4342/head * [new ref] refs/pull/4343/head -> refs/pull/4343/head * [new ref] refs/pull/4348/head -> refs/pull/4348/head * [new ref] refs/pull/4348/merge -> refs/pull/4348/merge * [new ref] refs/pull/4350/head -> refs/pull/4350/head * [new ref] refs/pull/4351/head -> refs/pull/4351/head * [new ref] refs/pull/4352/head -> refs/pull/4352/head * [new ref] refs/pull/4353/head -> refs/pull/4353/head * [new ref] refs/pull/4354/head -> refs/pull/4354/head * [new ref] refs/pull/4356/head -> refs/pull/4356/head * [new ref] refs/pull/4357/head -> refs/pull/4357/head * [new ref] refs/pull/4358/head -> refs/pull/4358/head * [new ref] refs/pull/4359/head -> refs/pull/4359/head * [new ref] refs/pull/4360/head -> refs/pull/4360/head * [new ref] refs/pull/4360/merge -> refs/pull/4360/merge * [new ref] refs/pull/4361/head -> refs/pull/4361/head * [new ref] refs/pull/4362/head -> refs/pull/4362/head * [new ref] refs/pull/4363/head -> refs/pull/4363/head * [new ref] refs/pull/4364/head -> refs/pull/4364/head * [new ref] refs/pull/4366/head -> refs/pull/4366/head * [new ref] refs/pull/4366/merge -> refs/pull/4366/merge * [new ref] refs/pull/4368/head -> refs/pull/4368/head * [new ref] refs/pull/4369/head -> refs/pull/4369/head * [new ref] refs/pull/4369/merge -> refs/pull/4369/merge * [new ref] refs/pull/4370/head -> refs/pull/4370/head * [new ref] refs/pull/4371/head -> refs/pull/4371/head * [new ref] refs/pull/4371/merge -> refs/pull/4371/merge * [new ref] refs/pull/4372/head -> refs/pull/4372/head * [new ref] refs/pull/4372/merge -> refs/pull/4372/merge * [new ref] refs/pull/4373/head -> refs/pull/4373/head * [new ref] refs/pull/4373/merge -> refs/pull/4373/merge * [new ref] refs/pull/4374/head -> refs/pull/4374/head * [new ref] refs/pull/4375/head -> refs/pull/4375/head * [new ref] refs/pull/4375/merge -> refs/pull/4375/merge * [new ref] refs/pull/4376/head -> refs/pull/4376/head * [new ref] refs/pull/4376/merge -> refs/pull/4376/merge * [new ref] refs/pull/4377/head -> refs/pull/4377/head * [new ref] refs/pull/4377/merge -> refs/pull/4377/merge * [new ref] refs/pull/4381/head -> refs/pull/4381/head * [new ref] refs/pull/4382/head -> refs/pull/4382/head * [new ref] refs/pull/4385/head -> refs/pull/4385/head * [new ref] refs/pull/4385/merge -> refs/pull/4385/merge * [new ref] refs/pull/4386/head -> refs/pull/4386/head * [new ref] refs/pull/4386/merge -> refs/pull/4386/merge * [new ref] refs/pull/4387/head -> refs/pull/4387/head * [new ref] refs/pull/4388/head -> refs/pull/4388/head * [new ref] refs/pull/4388/merge -> refs/pull/4388/merge * [new ref] refs/pull/4389/head -> refs/pull/4389/head * [new ref] refs/pull/4389/merge -> refs/pull/4389/merge * [new ref] refs/pull/4390/head -> refs/pull/4390/head * [new ref] refs/pull/4393/head -> refs/pull/4393/head * [new ref] refs/pull/4393/merge -> refs/pull/4393/merge * [new ref] refs/pull/4394/head -> refs/pull/4394/head * [new ref] refs/pull/4395/head -> refs/pull/4395/head * [new ref] refs/pull/4396/head -> refs/pull/4396/head * [new ref] refs/pull/4397/head -> refs/pull/4397/head * [new ref] refs/pull/4397/merge -> refs/pull/4397/merge * [new ref] refs/pull/4400/head -> refs/pull/4400/head * [new ref] refs/pull/4400/merge -> refs/pull/4400/merge * [new ref] refs/pull/4401/head -> refs/pull/4401/head * [new ref] refs/pull/4401/merge -> refs/pull/4401/merge * [new ref] origin/feature/3382 -> origin/feature/3382 * [new tag] mimir-2.6.0 -> mimir-2.6.0 * [new tag] mimir-2.6.0-rc.0 -> mimir-2.6.0-rc.0 * [new tag] mimir-distributed-4.1.0 -> mimir-distributed-4.1.0 * [new tag] mimir-distributed-4.1.0-weekly.222 -> mimir-distributed-4.1.0-weekly.222 * [new tag] mimir-distributed-4.1.0-weekly.223 -> mimir-distributed-4.1.0-weekly.223 * [new tag] mimir-distributed-4.1.0-weekly.224 -> mimir-distributed-4.1.0-weekly.224 * [new tag] mimir-distributed-4.1.0-weekly.225 -> mimir-distributed-4.1.0-weekly.225 * [new tag] mimir-distributed-4.2.0 -> mimir-distributed-4.2.0 * [new tag] mimir-distributed-4.3.0-weekly.227 -> mimir-distributed-4.3.0-weekly.227 * [new tag] mimir-distributed-4.3.0-weekly.228 -> mimir-distributed-4.3.0-weekly.228  -> Found systemd.service  -> Found sysusers.conf  -> Found tmpfiles.conf  -> Found config.yaml ==> Validating source files with b2sums... mimir ... Skipped systemd.service ... Passed sysusers.conf ... Passed tmpfiles.conf ... Passed config.yaml ... Passed ==> Making package: mimir 2.6.0-1 (Tue Mar 7 06:26:21 2023) ==> Checking runtime dependencies... ==> Checking buildtime dependencies... ==> Installing missing dependencies... [?25lresolving dependencies... looking for conflicting packages... Package (5) New Version Net Change extra/perl-error 0.17029-4 0.04 MiB extra/perl-mailtools 2.21-6 0.11 MiB extra/perl-timedate 2.33-4 0.08 MiB extra/git 2.39.2-1 33.03 MiB community/go 2:1.19.5-1 434.82 MiB Total Installed Size: 468.07 MiB :: Proceed with installation? [Y/n] checking keyring... checking package integrity... loading package files... checking for file conflicts... :: Processing package changes... installing perl-error... installing perl-timedate... installing perl-mailtools... installing git... Optional dependencies for git tk: gitk and git gui perl-libwww: git svn perl-term-readkey: git svn and interactive.singlekey setting perl-io-socket-ssl: git send-email TLS support perl-authen-sasl: git send-email TLS support perl-mediawiki-api: git mediawiki support perl-datetime-format-iso8601: git mediawiki support perl-lwp-protocol-https: git mediawiki https support perl-cgi: gitweb (web interface) support python: git svn & git p4 subversion: git svn org.freedesktop.secrets: keyring credential helper libsecret: libsecret credential helper [installed] installing go... :: Running post-transaction hooks... (1/1) Warn about old perl modules [?25h==> Retrieving sources...  -> Found systemd.service  -> Found sysusers.conf  -> Found tmpfiles.conf  -> Found config.yaml ==> WARNING: Skipping all source file integrity checks. ==> Extracting sources...  -> Creating working copy of mimir git repo... Cloning into 'mimir'... done. Updating files: 18% (1574/8726) Updating files: 19% (1658/8726) Updating files: 20% (1746/8726) Updating files: 21% (1833/8726) Updating files: 22% (1920/8726) Updating files: 23% (2007/8726) Updating files: 24% (2095/8726) Updating files: 25% (2182/8726) Updating files: 26% (2269/8726) Updating files: 27% (2357/8726) Updating files: 28% (2444/8726) Updating files: 29% (2531/8726) Updating files: 29% (2566/8726) Updating files: 30% (2618/8726) Updating files: 31% (2706/8726) Updating files: 32% (2793/8726) Updating files: 33% (2880/8726) Updating files: 34% (2967/8726) Updating files: 35% (3055/8726) Updating files: 36% (3142/8726) Updating files: 37% (3229/8726) Updating files: 38% (3316/8726) Updating files: 39% (3404/8726) Updating files: 40% (3491/8726) Updating files: 41% (3578/8726) Updating files: 42% (3665/8726) Updating files: 43% (3753/8726) Updating files: 44% (3840/8726) Updating files: 45% (3927/8726) Updating files: 46% (4014/8726) Updating files: 47% (4102/8726) Updating files: 47% (4169/8726) Updating files: 48% (4189/8726) Updating files: 49% (4276/8726) Updating files: 50% (4363/8726) Updating files: 51% (4451/8726) Updating files: 52% (4538/8726) Updating files: 53% (4625/8726) Updating files: 54% (4713/8726) Updating files: 55% (4800/8726) Updating files: 56% (4887/8726) Updating files: 57% (4974/8726) Updating files: 58% (5062/8726) Updating files: 59% (5149/8726) Updating files: 60% (5236/8726) Updating files: 61% (5323/8726) Updating files: 62% (5411/8726) Updating files: 63% (5498/8726) Updating files: 64% (5585/8726) Updating files: 65% (5672/8726) Updating files: 65% (5684/8726) Updating files: 66% (5760/8726) Updating files: 67% (5847/8726) Updating files: 68% (5934/8726) Updating files: 69% (6021/8726) Updating files: 70% (6109/8726) Updating files: 71% (6196/8726) Updating files: 72% (6283/8726) Updating files: 73% (6370/8726) Updating files: 74% (6458/8726) Updating files: 75% (6545/8726) Updating files: 76% (6632/8726) Updating files: 77% (6720/8726) Updating files: 78% (6807/8726) Updating files: 79% (6894/8726) Updating files: 80% (6981/8726) Updating files: 81% (7069/8726) Updating files: 82% (7156/8726) Updating files: 83% (7243/8726) Updating files: 83% (7319/8726) Updating files: 84% (7330/8726) Updating files: 85% (7418/8726) Updating files: 86% (7505/8726) Updating files: 87% (7592/8726) Updating files: 88% (7679/8726) Updating files: 89% (7767/8726) Updating files: 90% (7854/8726) Updating files: 91% (7941/8726) Updating files: 92% (8028/8726) Updating files: 93% (8116/8726) Updating files: 94% (8203/8726) Updating files: 95% (8290/8726) Updating files: 96% (8377/8726) Updating files: 97% (8465/8726) Updating files: 97% (8532/8726) Updating files: 98% (8552/8726) Updating files: 99% (8639/8726) Updating files: 100% (8726/8726) Updating files: 100% (8726/8726), done. Switched to a new branch 'makepkg' ==> Starting prepare()... ==> Starting pkgver()... ==> Starting build()... all modules verified go: downloading github.com/prometheus/client_golang v1.14.0 go: downloading gopkg.in/alecthomas/kingpin.v2 v2.2.6 go: downloading github.com/go-kit/log v0.2.1 go: downloading github.com/weaveworks/common v0.0.0-20221201103051-7c2720a9024d go: downloading github.com/grafana/dskit v0.0.0-20230109170026-7242706251b9 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/colega/go-yaml-yaml v0.0.0-20220720105220-255a8d16d094 go: downloading github.com/thanos-io/objstore v0.0.0-20221213124554-e4d8ba6bc6f3 go: downloading golang.org/x/exp v0.0.0-20221212164502-fae10dda9338 go: downloading github.com/google/go-github/v32 v32.1.0 go: downloading github.com/sirupsen/logrus v1.9.0 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/golang/snappy v0.0.4 go: downloading github.com/grafana-tools/sdk v0.0.0-20211220201350-966b3088eec9 go: downloading github.com/prometheus/alertmanager v0.25.0-rc.0.0.20221216141313-26cbd6bd862e go: downloading github.com/prometheus/common v0.39.0 go: downloading github.com/grafana/mimir-prometheus v0.0.0-20230119151144-44904a663c20 go: downloading github.com/gorilla/mux v1.8.0 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading go.uber.org/atomic v1.10.0 go: downloading github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751 go: downloading github.com/alecthomas/units v0.0.0-20211218093645-b94a6e3cc137 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/cespare/xxhash/v2 v2.2.0 go: downloading github.com/golang/protobuf v1.5.2 go: downloading github.com/prometheus/client_model v0.3.0 go: downloading github.com/prometheus/procfs v0.8.0 go: downloading github.com/cespare/xxhash v1.1.0 go: downloading google.golang.org/protobuf v1.28.1 go: downloading github.com/opentracing-contrib/go-stdlib v1.0.0 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading go.opentelemetry.io/otel v1.11.2 go: downloading go.opentelemetry.io/otel/trace v1.11.2 go: downloading google.golang.org/grpc v1.45.0 go: downloading github.com/weaveworks/promrus v1.2.0 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.3.0 go: downloading github.com/opentracing-contrib/go-grpc v0.0.0-20210225150812-73cb765af46e go: downloading github.com/prometheus/exporter-toolkit v0.8.2 go: downloading golang.org/x/net v0.5.0 go: downloading golang.org/x/sync v0.1.0 go: downloading golang.org/x/time v0.3.0 go: downloading github.com/go-logfmt/logfmt v0.5.1 go: downloading github.com/grafana/regexp v0.0.0-20221005093135-b4c2bcb0a4b6 go: downloading github.com/minio/minio-go/v7 v7.0.45 go: downloading github.com/stretchr/testify v1.8.1 go: downloading github.com/oklog/ulid v1.3.1 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/efficientgo/core v1.0.0-rc.0.0.20221201130417-ba593f67d2a4 go: downloading golang.org/x/sys v0.4.0 go: downloading github.com/alecthomas/chroma v0.10.0 go: downloading github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db go: downloading github.com/google/go-querystring v1.1.0 go: downloading golang.org/x/crypto v0.3.0 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/gosimple/slug v1.1.1 go: downloading github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f go: downloading golang.org/x/oauth2 v0.3.0 go: downloading github.com/prometheus/common/sigv4 v0.1.0 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.37.0 go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.4 go: downloading github.com/gogo/status v1.1.1 go: downloading github.com/armon/go-metrics v0.4.0 go: downloading github.com/hashicorp/go-sockaddr v1.0.2 go: downloading github.com/grafana/memberlist v0.3.1-0.20220714140823-09ffed8adbbe go: downloading github.com/felixge/fgprof v0.9.2 go: downloading github.com/segmentio/fasthash v0.0.0-20180216231524-a72b379d632e go: downloading github.com/spf13/afero v1.8.2 go: downloading github.com/google/go-cmp v0.5.9 go: downloading github.com/google/uuid v1.3.0 go: downloading github.com/edsrzf/mmap-go v1.1.0 go: downloading github.com/sercand/kuberesolver v2.4.0+incompatible go: downloading github.com/go-logr/logr v1.2.3 go: downloading github.com/coreos/go-systemd/v22 v22.4.0 go: downloading github.com/gogo/googleapis v1.4.1 go: downloading github.com/felixge/httpsnoop v1.0.3 go: downloading google.golang.org/genproto v0.0.0-20221207170731-23e4bf6bdc37 go: downloading github.com/davecgh/go-spew v1.1.1 go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading github.com/stretchr/objx v0.5.0 go: downloading github.com/aws/aws-sdk-go-v2 v1.16.0 go: downloading github.com/aws/aws-sdk-go-v2/config v1.15.1 go: downloading github.com/facette/natsort v0.0.0-20181210072756-2cd4dd1e2dcb go: downloading github.com/go-redis/redis/v8 v8.11.5 go: downloading github.com/grafana/gomemcache v0.0.0-20230105173749-11f792309e1f go: downloading github.com/hashicorp/golang-lru v0.6.0 go: downloading github.com/dennwc/varint v1.0.0 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/rainycape/unidecode v0.0.0-20150907023854-cb7f23ec59be go: downloading github.com/jpillora/backoff v1.0.0 go: downloading github.com/aws/aws-sdk-go v1.44.159 go: downloading github.com/miekg/dns v1.1.50 go: downloading go.opentelemetry.io/otel/metric v0.34.0 go: downloading github.com/hashicorp/consul/api v1.18.0 go: downloading github.com/hashicorp/go-cleanhttp v0.5.2 go: downloading go.etcd.io/etcd/api/v3 v3.5.4 go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.4 go: downloading go.etcd.io/etcd/client/v3 v3.5.4 go: downloading github.com/hashicorp/go-immutable-radix v1.2.0 go: downloading github.com/go-openapi/swag v0.22.3 go: downloading github.com/oklog/run v1.1.0 go: downloading github.com/cenkalti/backoff/v4 v4.2.0 go: downloading gopkg.in/telebot.v3 v3.1.2 go: downloading github.com/benbjohnson/clock v1.3.0 go: downloading github.com/gofrs/uuid v4.3.1+incompatible go: downloading golang.org/x/text v0.6.0 go: downloading github.com/julienschmidt/httprouter v1.3.0 go: downloading github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/prometheusremotewrite v0.69.0 go: downloading go.opentelemetry.io/collector/pdata v1.0.0-rc3.0.20230109164642-7d168dd20efd go: downloading go.opentelemetry.io/collector v0.57.2 go: downloading go.uber.org/multierr v1.9.0 go: downloading github.com/google/btree v1.0.1 go: downloading github.com/hashicorp/go-msgpack v0.5.5 go: downloading github.com/hashicorp/go-multierror v1.1.1 go: downloading github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529 go: downloading github.com/google/pprof v0.0.0-20221212185716-aee1124e3a93 go: downloading github.com/go-openapi/strfmt v0.21.3 go: downloading github.com/dustin/go-humanize v1.0.0 go: downloading go.uber.org/goleak v1.2.0 go: downloading github.com/go-logr/stdr v1.2.2 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.2.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.2.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v0.5.1 go: downloading cloud.google.com/go/storage v1.27.0 go: downloading google.golang.org/api v0.104.0 go: downloading github.com/ncw/swift v1.0.53 go: downloading github.com/klauspost/compress v1.15.9 go: downloading cloud.google.com/go v0.105.0 go: downloading github.com/minio/md5-simd v1.1.2 go: downloading github.com/minio/sha256-simd v1.0.0 go: downloading gopkg.in/ini.v1 v1.67.0 go: downloading github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f go: downloading github.com/aws/smithy-go v1.11.1 go: downloading github.com/aws/aws-sdk-go-v2/credentials v1.11.0 go: downloading github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.12.1 go: downloading github.com/aws/aws-sdk-go-v2/internal/ini v1.3.8 go: downloading github.com/aws/aws-sdk-go-v2/service/sso v1.11.1 go: downloading github.com/aws/aws-sdk-go-v2/service/sts v1.16.1 go: downloading github.com/dlclark/regexp2 v1.4.0 go: downloading github.com/hashicorp/go-hclog v0.12.2 go: downloading github.com/hashicorp/go-rootcerts v1.0.2 go: downloading github.com/hashicorp/serf v0.10.1 go: downloading github.com/mitchellh/mapstructure v1.5.0 go: downloading go.uber.org/zap v1.21.0 go: downloading github.com/go-openapi/analysis v0.21.4 go: downloading github.com/go-openapi/loads v0.21.2 go: downloading github.com/go-openapi/runtime v0.25.0 go: downloading github.com/rs/cors v1.8.2 go: downloading github.com/go-openapi/errors v0.20.3 go: downloading github.com/go-openapi/validate v0.22.0 go: downloading github.com/mailru/easyjson v0.7.7 go: downloading github.com/shurcooL/vfsgen v0.0.0-20200824052919-0d455de96546 go: downloading github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/prometheus v0.69.0 go: downloading go.opentelemetry.io/collector/semconv v0.69.0 go: downloading github.com/hashicorp/errwrap v1.1.0 go: downloading github.com/asaskevich/govalidator v0.0.0-20210307081110-f21760c49a8d go: downloading go.mongodb.org/mongo-driver v1.11.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.1.1 go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v0.7.0 go: downloading cloud.google.com/go/compute/metadata v0.2.2 go: downloading cloud.google.com/go/iam v0.8.0 go: downloading cloud.google.com/go/compute v1.13.0 go: downloading github.com/googleapis/gax-go/v2 v2.7.0 go: downloading github.com/rs/xid v1.4.0 go: downloading github.com/klauspost/cpuid/v2 v2.1.1 go: downloading github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.7 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.1 go: downloading github.com/fatih/color v1.13.0 go: downloading github.com/mattn/go-isatty v0.0.14 go: downloading github.com/go-openapi/jsonpointer v0.19.5 go: downloading github.com/go-openapi/spec v0.20.7 go: downloading github.com/jessevdk/go-flags v1.5.0 go: downloading github.com/jmespath/go-jmespath v0.4.0 go: downloading github.com/coreos/go-semver v0.3.0 go: downloading github.com/josharian/intern v1.0.0 go: downloading github.com/shurcooL/httpfs v0.0.0-20190707220628-8d4bc4ba7749 go: downloading go.opentelemetry.io/collector/featuregate v0.69.0 go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 go: downloading go.opencensus.io v0.24.0 go: downloading golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 go: downloading github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.1 go: downloading github.com/mattn/go-colorable v0.1.12 go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/golang-jwt/jwt/v4 v4.4.3 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.2.0 go: downloading github.com/go-openapi/jsonreference v0.20.0 internal/unsafeheader internal/goexperiment internal/goarch internal/cpu internal/goos runtime/internal/atomic internal/abi runtime/internal/math runtime/internal/sys runtime/internal/syscall internal/bytealg internal/race sync/atomic unicode unicode/utf8 runtime internal/itoa encoding math/bits unicode/utf16 container/list math crypto/internal/subtle crypto/subtle crypto/internal/boring/sig vendor/golang.org/x/crypto/cryptobyte/asn1 internal/nettrace vendor/golang.org/x/crypto/internal/subtle google.golang.org/protobuf/internal/flags google.golang.org/protobuf/internal/set github.com/Azure/azure-sdk-for-go/sdk/azcore/cloud github.com/Azure/azure-sdk-for-go/sdk/azcore/to github.com/Azure/azure-sdk-for-go/sdk/internal/errorinfo github.com/AzureAD/microsoft-authentication-library-for-go/apps/cache github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/exported github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops/internal/grant github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/version github.com/googleapis/gax-go/v2/internal google.golang.org/grpc/attributes google.golang.org/grpc/serviceconfig github.com/golang/groupcache/lru go.opencensus.io go.opencensus.io/trace/internal golang.org/x/xerrors/internal cloud.google.com/go/storage/internal go.opencensus.io/internal/tagencoding github.com/google/go-cmp/cmp/internal/flags github.com/weaveworks/common/errors github.com/uber/jaeger-client-go/internal/baggage github.com/uber/jaeger-client-go/internal/reporterstats github.com/uber/jaeger-client-go/internal/throttler github.com/aws/aws-sdk-go-v2/internal/sdkio github.com/aws/aws-sdk-go-v2 github.com/dgryski/go-rendezvous golang.org/x/net/internal/iana github.com/grafana/dskit/internal/slices github.com/prometheus/prometheus/model/value golang.org/x/exp/constraints github.com/dennwc/varint github.com/aws/aws-sdk-go/aws/client/metadata golang.org/x/exp/slices github.com/aws/aws-sdk-go/internal/sdkmath github.com/prometheus/prometheus/tsdb/goversion github.com/grafana/dskit/internal/math go.mongodb.org/mongo-driver/bson/bsonoptions go.mongodb.org/mongo-driver/bson/bsontype go.opentelemetry.io/collector/semconv/v1.6.1 go.opentelemetry.io/otel/internal go.opentelemetry.io/otel/metric/unit github.com/segmentio/fasthash/fnv1a image/color go.opentelemetry.io/otel/metric/instrument runtime/metrics sync internal/reflectlite math/rand internal/singleflight internal/testlog google.golang.org/protobuf/internal/pragma runtime/cgo internal/sysinfo google.golang.org/grpc/internal/buffer errors google.golang.org/grpc/internal/grpcsync sort github.com/aws/aws-sdk-go-v2/internal/sync/singleflight io internal/oserror syscall strconv bytes internal/safefilepath path reflect strings time internal/syscall/unix internal/syscall/execenv bufio regexp/syntax hash hash/crc32 crypto crypto/internal/randutil crypto/internal/nistec/fiat regexp context internal/poll io/fs crypto/rc4 os internal/fmtsort encoding/binary vendor/golang.org/x/net/dns/dnsmessage encoding/base64 crypto/cipher fmt crypto/internal/nistec crypto/internal/boring crypto/des crypto/aes crypto/sha512 crypto/internal/edwards25519/field crypto/hmac crypto/md5 crypto/internal/edwards25519 crypto/sha1 flag encoding/json github.com/go-logfmt/logfmt log github.com/alecthomas/units github.com/pkg/errors compress/flate crypto/sha256 math/big encoding/hex encoding/pem internal/godebug internal/intern net/netip github.com/go-kit/log compress/gzip net/url github.com/go-kit/log/level path/filepath net vendor/golang.org/x/crypto/chacha20 vendor/golang.org/x/crypto/internal/poly1305 vendor/golang.org/x/crypto/curve25519/internal/field vendor/golang.org/x/crypto/chacha20poly1305 vendor/golang.org/x/crypto/hkdf vendor/golang.org/x/text/transform vendor/golang.org/x/crypto/curve25519 vendor/golang.org/x/text/unicode/bidi vendor/golang.org/x/text/unicode/norm crypto/rand crypto/elliptic crypto/internal/boring/bbig encoding/asn1 crypto/ed25519 crypto/rsa crypto/dsa vendor/golang.org/x/crypto/cryptobyte crypto/x509/pkix vendor/golang.org/x/text/secure/bidirule vendor/golang.org/x/net/http2/hpack vendor/golang.org/x/net/idna crypto/ecdsa mime mime/quotedprintable net/http/internal net/http/internal/ascii github.com/beorn7/perks/quantile github.com/cespare/xxhash/v2 hash/fnv go/token google.golang.org/protobuf/internal/detrand io/ioutil google.golang.org/protobuf/internal/errors google.golang.org/protobuf/internal/version github.com/prometheus/common/model github.com/prometheus/common/internal/bitbucket.org/ww/goautoneg google.golang.org/protobuf/encoding/protowire github.com/prometheus/procfs/internal/fs github.com/prometheus/procfs/internal/util google.golang.org/protobuf/reflect/protoreflect golang.org/x/sys/unix runtime/debug github.com/efficientgo/core/errors github.com/efficientgo/core/logerrcapture github.com/davecgh/go-spew/spew google.golang.org/protobuf/internal/encoding/messageset google.golang.org/protobuf/internal/strs google.golang.org/protobuf/internal/encoding/text google.golang.org/protobuf/internal/genid google.golang.org/protobuf/internal/order google.golang.org/protobuf/reflect/protoregistry crypto/x509 net/textproto vendor/golang.org/x/net/http/httpproxy vendor/golang.org/x/net/http/httpguts mime/multipart google.golang.org/protobuf/runtime/protoiface google.golang.org/protobuf/proto google.golang.org/protobuf/internal/encoding/defval google.golang.org/protobuf/internal/descfmt google.golang.org/protobuf/internal/descopts github.com/prometheus/procfs github.com/efficientgo/core/testutil/internal runtime/trace testing google.golang.org/protobuf/encoding/prototext crypto/tls google.golang.org/protobuf/internal/filedesc github.com/efficientgo/core/testutil github.com/opentracing/opentracing-go/log golang.org/x/sync/errgroup golang.org/x/text/transform golang.org/x/text/unicode/bidi golang.org/x/text/unicode/norm google.golang.org/protobuf/internal/encoding/tag golang.org/x/text/secure/bidirule google.golang.org/protobuf/internal/impl github.com/Azure/azure-sdk-for-go/sdk/azcore/tracing encoding/xml net/http/httptrace net/http golang.org/x/net/idna github.com/Azure/azure-sdk-for-go/sdk/internal/log github.com/Azure/azure-sdk-for-go/sdk/azcore/log github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/log github.com/Azure/azure-sdk-for-go/sdk/internal/diag github.com/Azure/azure-sdk-for-go/sdk/internal/temporal github.com/Azure/azure-sdk-for-go/sdk/internal/uuid golang.org/x/net/http/httpguts github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/json github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/json/types/time github.com/kylelemons/godebug/diff github.com/kylelemons/godebug/pretty database/sql/driver github.com/golang-jwt/jwt/v4 github.com/google/uuid os/exec golang.org/x/crypto/pkcs12/internal/rc2 golang.org/x/crypto/pkcs12 github.com/pkg/browser gopkg.in/yaml.v2 gopkg.in/yaml.v3 google.golang.org/protobuf/internal/filetype google.golang.org/protobuf/runtime/protoimpl google.golang.org/protobuf/types/descriptorpb google.golang.org/protobuf/types/known/timestamppb github.com/golang/protobuf/ptypes/timestamp github.com/efficientgo/core/merrors google.golang.org/genproto/googleapis/rpc/code github.com/efficientgo/core/errcapture google.golang.org/protobuf/types/known/anypb google.golang.org/api/internal/third_party/uritemplates google.golang.org/protobuf/reflect/protodesc google.golang.org/protobuf/types/known/durationpb github.com/googleapis/gax-go/v2/apierror/internal/proto google.golang.org/genproto/googleapis/rpc/errdetails google.golang.org/genproto/googleapis/rpc/status google.golang.org/grpc/codes github.com/golang/protobuf/ptypes/any github.com/golang/protobuf/ptypes/duration google.golang.org/protobuf/internal/encoding/json golang.org/x/net/internal/timeseries expvar github.com/golang/protobuf/proto github.com/opentracing/opentracing-go github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared github.com/AzureAD/microsoft-authentication-library-for-go/apps/errors github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported github.com/thanos-io/objstore/tracing github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops/authority github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/shared github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops/internal/comm github.com/Azure/azure-sdk-for-go/sdk/azcore/policy github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/pollers github.com/Azure/azure-sdk-for-go/sdk/azcore github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/pollers/async github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/pollers/body github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/pollers/loc github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/pollers/op github.com/Azure/azure-sdk-for-go/sdk/azcore/streaming github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops/wstrust/defs github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/local github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime github.com/prometheus/client_model/go github.com/matttproud/golang_protobuf_extensions/pbutil github.com/thanos-io/objstore/exthttp github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops/wstrust cloud.google.com/go/compute/metadata github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops/accesstokens github.com/prometheus/client_golang/prometheus/internal github.com/prometheus/common/expfmt github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/generated google.golang.org/api/googleapi github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth github.com/golang/protobuf/ptypes github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/base/internal/storage google.golang.org/grpc/internal/status github.com/prometheus/client_golang/prometheus google.golang.org/grpc/status google.golang.org/protobuf/encoding/protojson github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/base github.com/AzureAD/microsoft-authentication-library-for-go/apps/confidential github.com/AzureAD/microsoft-authentication-library-for-go/apps/public github.com/googleapis/gax-go/v2/apierror github.com/Azure/azure-sdk-for-go/sdk/azidentity html text/template/parse github.com/prometheus/client_golang/prometheus/promauto text/tabwriter github.com/grafana/dskit/flagext github.com/thanos-io/objstore google.golang.org/grpc/backoff google.golang.org/grpc/internal/grpclog google.golang.org/grpc/grpclog google.golang.org/grpc/connectivity text/template google.golang.org/grpc/internal/credentials github.com/thanos-io/objstore/providers/filesystem google.golang.org/grpc/credentials github.com/grafana/mimir/pkg/storage/bucket/filesystem google.golang.org/grpc/metadata github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/shared github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/bloberror google.golang.org/grpc/resolver google.golang.org/grpc/internal/grpcrand google.golang.org/grpc/internal google.golang.org/grpc/credentials/insecure github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/exported google.golang.org/grpc/balancer google.golang.org/grpc/encoding google.golang.org/grpc/encoding/proto google.golang.org/grpc/balancer/base google.golang.org/grpc/internal/backoff github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/base github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/sas html/template google.golang.org/grpc/balancer/roundrobin google.golang.org/grpc/internal/balancerload google.golang.org/grpc/binarylog/grpc_binarylog_v1 google.golang.org/grpc/internal/grpcutil google.golang.org/grpc/internal/channelz github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blob google.golang.org/grpc/internal/binarylog golang.org/x/net/trace google.golang.org/grpc/internal/serviceconfig github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blockblob github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/appendblob github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/pageblob google.golang.org/grpc/internal/resolver google.golang.org/grpc/balancer/grpclb/state google.golang.org/grpc/internal/envconfig google.golang.org/grpc/internal/resolver/passthrough google.golang.org/grpc/internal/resolver/dns github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/container google.golang.org/grpc/internal/transport/networktype golang.org/x/net/http2/hpack google.golang.org/grpc/internal/resolver/unix google.golang.org/grpc/internal/metadata google.golang.org/grpc/internal/syscall google.golang.org/grpc/keepalive google.golang.org/grpc/peer google.golang.org/grpc/stats golang.org/x/net/http2 google.golang.org/grpc/tap net/http/httputil google.golang.org/genproto/googleapis/api github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/service google.golang.org/genproto/googleapis/api/annotations google.golang.org/genproto/googleapis/type/expr github.com/Azure/azure-sdk-for-go/sdk/storage/azblob google.golang.org/protobuf/types/known/fieldmaskpb cloud.google.com/go/internal/optional go.opencensus.io/internal go.opencensus.io/trace/tracestate golang.org/x/xerrors go.opencensus.io/trace github.com/thanos-io/objstore/providers/azure cloud.google.com/go/internal/version google.golang.org/protobuf/types/known/emptypb github.com/golang/protobuf/ptypes/empty cloud.google.com/go/internal/trace google.golang.org/genproto/googleapis/type/date github.com/grafana/mimir/pkg/storage/bucket/azure google.golang.org/genproto/protobuf/field_mask google.golang.org/api/iterator golang.org/x/net/context/ctxhttp golang.org/x/oauth2/jws golang.org/x/oauth2/internal os/user go.opencensus.io/resource go.opencensus.io/metric/metricdata golang.org/x/oauth2 runtime/pprof golang.org/x/oauth2/authhandler golang.org/x/oauth2/google/internal/externalaccount google.golang.org/grpc/internal/transport golang.org/x/oauth2/jwt google.golang.org/api/internal/impersonate go.opencensus.io/tag go.opencensus.io/metric/metricproducer go.opencensus.io/trace/propagation encoding/gob go.opencensus.io/stats/internal go.opencensus.io/stats go.opencensus.io/stats/view go.opencensus.io/plugin/ocgrpc golang.org/x/oauth2/google github.com/googleapis/enterprise-certificate-proxy/client/util github.com/google/go-cmp/cmp/internal/diff google.golang.org/grpc net/rpc github.com/google/go-cmp/cmp/internal/function github.com/google/go-cmp/cmp/internal/value google.golang.org/grpc/credentials/alts/internal google.golang.org/grpc/credentials/alts/internal/conn github.com/google/go-cmp/cmp google.golang.org/grpc/internal/googlecloud github.com/googleapis/enterprise-certificate-proxy/client google.golang.org/grpc/credentials/oauth google.golang.org/api/transport/cert go.opencensus.io/plugin/ochttp/propagation/b3 google.golang.org/api/googleapi/transport go.opencensus.io/plugin/ochttp google.golang.org/api/transport/http/internal/propagation github.com/prometheus/common/version github.com/gogo/protobuf/proto github.com/golang/snappy github.com/googleapis/gax-go/v2 cloud.google.com/go/iam/apiv1/iampb google.golang.org/api/internal cloud.google.com/go/internal google.golang.org/grpc/balancer/grpclb/grpc_lb_v1 google.golang.org/api/option google.golang.org/genproto/googleapis/iam/v1 google.golang.org/api/option/internaloption cloud.google.com/go/iam cloud.google.com/go/storage/internal/apiv2/stubs google.golang.org/api/transport/internal/dca google.golang.org/grpc/balancer/grpclb google.golang.org/grpc/credentials/alts/internal/proto/grpc_gcp google.golang.org/grpc/credentials/alts/internal/handshaker/service google.golang.org/api/internal/gensupport google.golang.org/grpc/credentials/alts/internal/authinfo google.golang.org/grpc/credentials/alts/internal/handshaker google.golang.org/grpc/credentials/alts google.golang.org/api/transport/http google.golang.org/grpc/credentials/google google.golang.org/api/iamcredentials/v1 google.golang.org/api/transport/grpc google.golang.org/api/storage/v1 google.golang.org/api/transport github.com/grafana/dskit/services cloud.google.com/go/storage/internal/apiv2 github.com/sirupsen/logrus golang.org/x/net/context github.com/gorilla/mux github.com/grpc-ecosystem/go-grpc-middleware github.com/weaveworks/promrus github.com/weaveworks/common/logging github.com/opentracing/opentracing-go/ext github.com/prometheus/client_golang/prometheus/promhttp github.com/opentracing-contrib/go-grpc github.com/coreos/go-systemd/v22/activation github.com/weaveworks/common/user github.com/jpillora/backoff github.com/grafana/dskit/tenant github.com/mwitkow/go-conntrack golang.org/x/oauth2/clientcredentials golang.org/x/crypto/blowfish github.com/prometheus/common/config cloud.google.com/go/storage golang.org/x/crypto/bcrypt github.com/gogo/protobuf/sortkeys github.com/gogo/protobuf/types github.com/gogo/protobuf/protoc-gen-gogo/descriptor github.com/prometheus/exporter-toolkit/web github.com/sercand/kuberesolver github.com/felixge/httpsnoop github.com/gogo/protobuf/gogoproto github.com/opentracing-contrib/go-stdlib/nethttp github.com/weaveworks/common/grpc github.com/uber/jaeger-client-go/internal/spanlog github.com/uber/jaeger-client-go/log hash/adler32 compress/zlib github.com/uber/jaeger-client-go/thrift github.com/uber/jaeger-lib/metrics github.com/thanos-io/objstore/providers/gcs go.uber.org/atomic github.com/gogo/googleapis/google/rpc github.com/uber/jaeger-lib/metrics/prometheus github.com/grafana/mimir/pkg/storage/bucket/gcs net/http/internal/testcert os/signal net/http/httptest github.com/weaveworks/common/signals golang.org/x/net/netutil internal/profile github.com/grafana/regexp/syntax github.com/uber/jaeger-client-go/thrift-gen/jaeger github.com/gogo/status github.com/weaveworks/common/httpgrpc github.com/uber/jaeger-client-go/thrift-gen/sampling github.com/uber/jaeger-client-go/thrift-gen/zipkincore github.com/uber/jaeger-client-go/thrift-gen/baggage net/http/pprof github.com/grafana/regexp github.com/modern-go/concurrent github.com/uber/jaeger-client-go/thrift-gen/agent github.com/modern-go/reflect2 golang.org/x/crypto/blake2b github.com/uber/jaeger-client-go/utils golang.org/x/crypto/argon2 github.com/prometheus/prometheus/model/labels github.com/aws/aws-sdk-go-v2/internal/rand github.com/uber/jaeger-client-go github.com/aws/aws-sdk-go-v2/internal/sdk github.com/aws/smithy-go/logging github.com/aws/smithy-go/middleware github.com/aws/smithy-go/ptr github.com/aws/smithy-go/rand github.com/aws/aws-sdk-go-v2/aws github.com/aws/smithy-go github.com/aws/smithy-go/time github.com/aws/smithy-go/transport/http/internal/io github.com/aws/smithy-go/transport/http github.com/json-iterator/go github.com/aws/aws-sdk-go-v2/credentials github.com/aws/aws-sdk-go-v2/aws/ratelimit github.com/aws/aws-sdk-go-v2/internal/timeconv github.com/aws/aws-sdk-go-v2/feature/ec2/imds/internal/config github.com/aws/smithy-go/io github.com/aws/aws-sdk-go-v2/credentials/processcreds github.com/aws/aws-sdk-go-v2/aws/middleware github.com/uber/jaeger-client-go/internal/baggage/remote github.com/uber/jaeger-client-go/internal/throttler/remote github.com/uber/jaeger-client-go/rpcmetrics github.com/uber/jaeger-client-go/transport github.com/aws/aws-sdk-go-v2/aws/transport/http github.com/aws/aws-sdk-go-v2/aws/retry github.com/uber/jaeger-client-go/config github.com/aws/aws-sdk-go-v2/aws/defaults github.com/aws/aws-sdk-go-v2/aws/protocol/restjson github.com/aws/aws-sdk-go-v2/internal/strings github.com/aws/aws-sdk-go-v2/aws/signer/internal/v4 github.com/aws/aws-sdk-go-v2/feature/ec2/imds github.com/weaveworks/common/tracing github.com/aws/aws-sdk-go-v2/credentials/endpointcreds/internal/client github.com/weaveworks/common/instrument github.com/aws/aws-sdk-go-v2/credentials/ec2rolecreds github.com/aws/aws-sdk-go-v2/credentials/endpointcreds github.com/weaveworks/common/middleware github.com/aws/smithy-go/encoding/httpbinding github.com/aws/aws-sdk-go-v2/internal/configsources github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 github.com/aws/aws-sdk-go-v2/service/sso/internal/endpoints github.com/aws/aws-sdk-go-v2/aws/signer/v4 github.com/aws/smithy-go/document github.com/aws/aws-sdk-go-v2/service/sso/types github.com/weaveworks/common/httpgrpc/server github.com/minio/minio-go/v7/pkg/encrypt github.com/aws/aws-sdk-go-v2/aws/protocol/query github.com/aws/aws-sdk-go-v2/service/sso github.com/aws/aws-sdk-go-v2/aws/protocol/xml github.com/aws/aws-sdk-go-v2/service/internal/presigned-url github.com/aws/aws-sdk-go-v2/service/sts/internal/endpoints github.com/aws/aws-sdk-go-v2/service/sts/types github.com/weaveworks/common/server github.com/aws/smithy-go/encoding github.com/aws/aws-sdk-go-v2/internal/ini github.com/aws/smithy-go/encoding/xml github.com/aws/aws-sdk-go-v2/service/sts github.com/aws/aws-sdk-go-v2/credentials/ssocreds github.com/grafana/mimir/pkg/util/log archive/tar github.com/klauspost/compress/s2 github.com/grafana/mimir/pkg/util github.com/minio/md5-simd github.com/minio/minio-go/v7/pkg/s3utils github.com/minio/minio-go/v7/pkg/signer gopkg.in/ini.v1 github.com/minio/minio-go/v7/pkg/lifecycle github.com/minio/minio-go/v7/pkg/set github.com/aws/aws-sdk-go-v2/credentials/stscreds github.com/minio/minio-go/v7/pkg/notification github.com/rs/xid github.com/aws/aws-sdk-go-v2/config github.com/minio/minio-go/v7/pkg/sse github.com/minio/minio-go/v7/pkg/replication github.com/minio/minio-go/v7/pkg/tags github.com/klauspost/cpuid/v2 github.com/minio/minio-go/v7/pkg/credentials embed net/http/cookiejar golang.org/x/net/publicsuffix github.com/ncw/swift github.com/pmezard/go-difflib/difflib github.com/minio/sha256-simd github.com/stretchr/objx github.com/stretchr/testify/assert github.com/minio/minio-go/v7 github.com/thanos-io/objstore/providers/swift github.com/cespare/xxhash github.com/grafana/mimir/pkg/storage/bucket/swift github.com/facette/natsort github.com/go-redis/redis/v8/internal/rand github.com/go-redis/redis/v8/internal/util github.com/go-redis/redis/v8/internal/hashtag github.com/go-redis/redis/v8/internal github.com/stretchr/testify/mock github.com/go-redis/redis/v8/internal/hscan github.com/go-redis/redis/v8/internal/proto github.com/grafana/dskit/crypto/tls github.com/go-redis/redis/v8/internal/pool github.com/grafana/dskit/dns/godns encoding/base32 golang.org/x/net/bpf golang.org/x/net/internal/socket github.com/go-redis/redis/v8 github.com/grafana/dskit/multierror github.com/grafana/dskit/gate github.com/grafana/dskit/spanlogger golang.org/x/net/ipv4 golang.org/x/net/ipv6 github.com/grafana/gomemcache/memcache github.com/thanos-io/objstore/providers/s3 github.com/miekg/dns github.com/hashicorp/golang-lru/simplelru github.com/grafana/dskit/backoff github.com/grafana/dskit/grpcencoding/snappy github.com/grafana/mimir/pkg/storage/bucket/s3 golang.org/x/time/rate google.golang.org/grpc/encoding/gzip github.com/grafana/mimir/pkg/storage/bucket github.com/grafana/dskit/grpcclient github.com/prometheus/prometheus/model/exemplar github.com/prometheus/prometheus/model/histogram github.com/prometheus/prometheus/prompb/io/prometheus/client github.com/prometheus/prometheus/util/jsonutil google.golang.org/grpc/health/grpc_health_v1 github.com/golang/groupcache/singleflight github.com/grafana/dskit/runutil github.com/oklog/ulid github.com/prometheus/prometheus/model/textparse github.com/prometheus/prometheus/model/relabel github.com/prometheus/prometheus/model/timestamp container/heap github.com/prometheus/prometheus/tsdb/chunkenc github.com/grafana/mimir/pkg/mimirpb github.com/prometheus/prometheus/model/metadata github.com/prometheus/prometheus/tsdb/errors github.com/prometheus/prometheus/tsdb/fileutil github.com/prometheus/prometheus/tsdb/chunks github.com/grafana/mimir/pkg/ingester/client github.com/prometheus/client_golang/prometheus/testutil/promlint github.com/grafana/dskit/dns/miekgdns github.com/prometheus/client_golang/prometheus/testutil github.com/grafana/dskit/dns github.com/stretchr/testify/require go.uber.org/goleak/internal/stack go.uber.org/goleak github.com/prometheus/prometheus/util/strutil github.com/aws/aws-sdk-go/aws/awserr github.com/aws/aws-sdk-go/internal/ini github.com/prometheus/prometheus/util/testutil github.com/prometheus/prometheus/tsdb/tsdbutil github.com/aws/aws-sdk-go/internal/shareddefaults github.com/aws/aws-sdk-go/internal/sync/singleflight github.com/aws/aws-sdk-go/aws/credentials github.com/prometheus/prometheus/storage github.com/aws/aws-sdk-go/aws/endpoints github.com/aws/aws-sdk-go/internal/sdkio github.com/jmespath/go-jmespath github.com/aws/aws-sdk-go/aws/awsutil github.com/prometheus/prometheus/promql/parser github.com/aws/aws-sdk-go/internal/sdkrand github.com/aws/aws-sdk-go/internal/strings github.com/aws/aws-sdk-go/aws/credentials/processcreds github.com/aws/aws-sdk-go/internal/sdkuri github.com/prometheus/prometheus/discovery/targetgroup github.com/prometheus/prometheus/discovery github.com/prometheus/prometheus/tsdb/encoding github.com/prometheus/prometheus/tsdb/hashcache github.com/prometheus/prometheus/tsdb/index github.com/prometheus/prometheus/tsdb/tombstones github.com/prometheus/prometheus/tsdb/record github.com/prometheus/prometheus/tsdb/wlog golang.org/x/sync/semaphore github.com/grafana/mimir/pkg/util/extprom github.com/grafana/mimir/pkg/storage/sharding github.com/grafana/mimir/pkg/util/spanlogger github.com/grafana/mimir/pkg/util/math github.com/grafana/dskit/cache github.com/grafana/mimir/pkg/storegateway/indexheader/encoding github.com/grafana/mimir/pkg/storegateway/indexheader/fileutil github.com/grafana/mimir/pkg/storegateway/indexheader/index github.com/grafana/dskit/grpcutil github.com/grafana/dskit/kv/codec github.com/hashicorp/go-cleanhttp github.com/mattn/go-isatty github.com/hashicorp/go-rootcerts github.com/mattn/go-colorable github.com/hashicorp/go-immutable-radix github.com/fatih/color github.com/hashicorp/go-hclog github.com/armon/go-metrics github.com/grafana/mimir/pkg/storage/tsdb/bucketcache github.com/grafana/mimir/pkg/storegateway/indexcache github.com/hashicorp/serf/coordinate github.com/mitchellh/mapstructure go.etcd.io/etcd/api/v3/authpb go.etcd.io/etcd/api/v3/membershippb go.etcd.io/etcd/api/v3/mvccpb go.uber.org/multierr go.etcd.io/etcd/api/v3/etcdserverpb github.com/hashicorp/consul/api go.uber.org/zap/buffer go.uber.org/zap/internal/bufferpool go.uber.org/zap/internal/color go.uber.org/zap/internal/exit go.uber.org/zap/zapcore go.uber.org/zap go.etcd.io/etcd/client/pkg/v3/fileutil github.com/grafana/dskit/kv/consul go.etcd.io/etcd/client/pkg/v3/tlsutil go.etcd.io/etcd/client/pkg/v3/transport go.etcd.io/etcd/api/v3/v3rpc/rpctypes github.com/coreos/go-semver/semver github.com/coreos/go-systemd/v22/journal go.etcd.io/etcd/api/v3/version go.etcd.io/etcd/client/pkg/v3/systemd go.etcd.io/etcd/client/pkg/v3/types go.etcd.io/etcd/client/pkg/v3/logutil go.etcd.io/etcd/client/v3/credentials go.etcd.io/etcd/client/v3/internal/endpoint google.golang.org/grpc/resolver/manual go.uber.org/zap/zapgrpc go.etcd.io/etcd/client/v3/internal/resolver github.com/prometheus/client_golang/prometheus/push github.com/hashicorp/go-sockaddr github.com/armon/go-metrics/prometheus compress/lzw github.com/google/btree github.com/hashicorp/go-msgpack/codec github.com/aws/aws-sdk-go/aws github.com/hashicorp/errwrap github.com/hashicorp/go-multierror github.com/sean-/seed github.com/aws/aws-sdk-go/aws/request github.com/grafana/dskit/modules go.etcd.io/etcd/client/v3 github.com/grafana/dskit/netutil github.com/grafana/dskit/ring/shard github.com/grafana/dskit/runtimeconfig github.com/grafana/dskit/concurrency github.com/aws/aws-sdk-go/aws/client github.com/aws/aws-sdk-go/private/protocol github.com/aws/aws-sdk-go/aws/corehandlers github.com/aws/aws-sdk-go/aws/csm github.com/aws/aws-sdk-go/aws/ec2metadata github.com/aws/aws-sdk-go/private/protocol/rest github.com/aws/aws-sdk-go/private/protocol/query/queryutil github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil github.com/aws/aws-sdk-go/private/protocol/json/jsonutil github.com/aws/aws-sdk-go/aws/signer/v4 github.com/aws/aws-sdk-go/private/protocol/query github.com/aws/aws-sdk-go/private/protocol/jsonrpc github.com/aws/aws-sdk-go/aws/credentials/ec2rolecreds github.com/aws/aws-sdk-go/service/sts github.com/aws/aws-sdk-go/private/protocol/restjson github.com/aws/aws-sdk-go/aws/credentials/endpointcreds github.com/grafana/dskit/kv/etcd github.com/aws/aws-sdk-go/service/sso github.com/aws/aws-sdk-go/aws/defaults github.com/hashicorp/memberlist github.com/prometheus/alertmanager/cluster/clusterpb github.com/aws/aws-sdk-go/service/sso/ssoiface github.com/aws/aws-sdk-go/service/sts/stsiface github.com/aws/aws-sdk-go/aws/credentials/ssocreds github.com/aws/aws-sdk-go/aws/credentials/stscreds github.com/grafana/mimir/pkg/alertmanager/alertmanagerpb github.com/grafana/mimir/pkg/alertmanager/alertspb github.com/aws/aws-sdk-go/aws/session github.com/grafana/mimir/pkg/alertmanager/alertstore/bucketclient github.com/prometheus/alertmanager/pkg/labels github.com/prometheus/alertmanager/timeinterval github.com/josharian/intern github.com/mailru/easyjson/jlexer github.com/prometheus/common/sigv4 github.com/mailru/easyjson/buffer github.com/prometheus/prometheus/config github.com/mailru/easyjson/jwriter github.com/prometheus/alertmanager/config github.com/grafana/dskit/kv/memberlist github.com/go-openapi/swag github.com/prometheus/prometheus/tsdb github.com/grafana/mimir/pkg/alertmanager/alertstore/local github.com/prometheus/alertmanager/api/metrics github.com/grafana/mimir/pkg/alertmanager/alertstore github.com/hashicorp/golang-lru github.com/grafana/dskit/kv github.com/cenkalti/backoff/v4 github.com/prometheus/alertmanager/cluster github.com/oklog/run github.com/prometheus/alertmanager/types github.com/grafana/dskit/ring github.com/prometheus/alertmanager/provider github.com/prometheus/alertmanager/store github.com/prometheus/alertmanager/inhibit github.com/prometheus/alertmanager/nflog/nflogpb github.com/benbjohnson/clock github.com/gofrs/uuid github.com/prometheus/alertmanager/nflog github.com/prometheus/alertmanager/silence/silencepb github.com/shurcooL/httpfs/vfsutil github.com/prometheus/alertmanager/silence github.com/shurcooL/vfsgen github.com/prometheus/alertmanager/asset golang.org/x/text/internal/tag github.com/grafana/dskit/ring/client golang.org/x/text/internal/language github.com/julienschmidt/httprouter github.com/go-openapi/analysis/internal/debug github.com/go-openapi/jsonpointer github.com/prometheus/common/route github.com/go-openapi/jsonreference/internal github.com/asaskevich/govalidator github.com/go-openapi/jsonreference github.com/go-openapi/spec golang.org/x/text/internal/language/compact golang.org/x/text/language github.com/grafana/mimir/pkg/storage/tsdb/metadata github.com/grafana/mimir/pkg/storage/tsdb/block github.com/go-openapi/errors golang.org/x/text/internal go.mongodb.org/mongo-driver/bson/primitive golang.org/x/text/cases github.com/go-openapi/analysis/internal/flatten/normalize github.com/go-openapi/analysis/internal/flatten/operations github.com/go-openapi/analysis/internal/flatten/replace github.com/go-openapi/analysis/internal/flatten/schutils github.com/go-openapi/analysis/internal/flatten/sortref github.com/grafana/mimir/pkg/storegateway/indexheader go.mongodb.org/mongo-driver/x/bsonx/bsoncore net/mail github.com/prometheus/alertmanager/template github.com/prometheus/alertmanager/notify encoding/csv github.com/go-openapi/runtime/logger github.com/grafana/mimir/pkg/storage/tsdb github.com/go-openapi/runtime/middleware/denco github.com/prometheus/alertmanager/dispatch github.com/go-openapi/runtime/middleware/header go.mongodb.org/mongo-driver/bson/bsonrw github.com/docker/go-units github.com/grafana/mimir/cmd/metaconvert github.com/go-openapi/runtime/flagext github.com/prometheus/alertmanager/api/v1 github.com/jessevdk/go-flags github.com/rs/cors github.com/grafana/mimir/pkg/util/net github.com/prometheus/alertmanager/notify/discord net/smtp go.mongodb.org/mongo-driver/bson/bsoncodec github.com/prometheus/alertmanager/notify/email github.com/prometheus/alertmanager/notify/opsgenie github.com/prometheus/alertmanager/notify/pagerduty github.com/prometheus/alertmanager/notify/pushover github.com/prometheus/alertmanager/notify/slack github.com/aws/aws-sdk-go/service/sns gopkg.in/telebot.v3 go.mongodb.org/mongo-driver/bson github.com/go-openapi/strfmt github.com/prometheus/alertmanager/notify/sns github.com/prometheus/alertmanager/notify/telegram github.com/prometheus/alertmanager/notify/victorops github.com/prometheus/alertmanager/notify/webex github.com/prometheus/alertmanager/notify/webhook github.com/prometheus/alertmanager/notify/wechat github.com/prometheus/alertmanager/provider/mem github.com/prometheus/alertmanager/ui github.com/google/pprof/profile github.com/grafana/mimir/pkg/storage/tsdb/bucketindex github.com/go-openapi/analysis github.com/go-openapi/runtime github.com/go-openapi/runtime/security github.com/go-openapi/loads github.com/grafana/mimir/pkg/ingester/activeseries github.com/felixge/fgprof github.com/go-openapi/runtime/middleware/untyped github.com/go-openapi/validate github.com/grafana/mimir/pkg/util/ephemeral github.com/grafana/mimir/pkg/util/extract github.com/grafana/mimir/pkg/util/globalerror github.com/grafana/dskit/limiter github.com/grafana/mimir/pkg/querier/stats github.com/grafana/mimir/pkg/util/validation github.com/grafana/mimir/pkg/util/httpgrpcutil go.opentelemetry.io/collector/featuregate go.opentelemetry.io/collector/pdata/internal/data go.opentelemetry.io/collector/pdata/internal/data/protogen/common/v1 github.com/grafana/mimir/pkg/compactor github.com/grafana/mimir/pkg/distributor/forwarding github.com/go-openapi/runtime/middleware github.com/prometheus/alertmanager/api/v2/models github.com/grafana/mimir/pkg/util/limiter github.com/prometheus/alertmanager/api/v2/restapi/operations/alert github.com/prometheus/alertmanager/api/v2/restapi/operations/alertgroup github.com/prometheus/alertmanager/api/v2/restapi/operations/general github.com/prometheus/alertmanager/api/v2/restapi/operations/receiver github.com/prometheus/alertmanager/api/v2/restapi/operations/silence go.opentelemetry.io/collector/pdata/internal/data/protogen/resource/v1 github.com/gogo/protobuf/jsonpb go.opentelemetry.io/collector/pdata/internal/data/protogen/logs/v1 github.com/prometheus/alertmanager/api/v2/restapi/operations go.opentelemetry.io/collector/pdata/internal/data/protogen/collector/logs/v1 go.opentelemetry.io/collector/pdata/internal/data/protogen/metrics/v1 github.com/prometheus/alertmanager/api/v2/restapi go.opentelemetry.io/collector/pdata/internal/data/protogen/trace/v1 github.com/prometheus/alertmanager/api/v2 go.opentelemetry.io/collector/pdata/internal/data/protogen/collector/trace/v1 github.com/grafana/mimir/pkg/alertmanager/merger github.com/prometheus/alertmanager/api go.opentelemetry.io/collector/pdata/internal/json go.opentelemetry.io/collector/pdata/internal/data/protogen/collector/metrics/v1 go.opentelemetry.io/collector/pdata/internal/otlp github.com/grafana/mimir/pkg/alertmanager github.com/prometheus/prometheus/prompb go.opentelemetry.io/collector/pdata/internal go.opentelemetry.io/collector/pdata/pcommon go.opentelemetry.io/collector/pdata/pmetric/internal/pmetricjson go.opentelemetry.io/collector/pdata/plog/internal/plogjson go.opentelemetry.io/collector/pdata/pmetric go.opentelemetry.io/collector/pdata/plog go.opentelemetry.io/collector/pdata/ptrace/internal/ptracejson go.opentelemetry.io/collector/pdata/ptrace github.com/prometheus/prometheus/util/osutil github.com/prometheus/prometheus/util/pool github.com/prometheus/prometheus/scrape github.com/weaveworks/common/mtime github.com/grafana/mimir/pkg/distributor/distributorpb github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/prometheus go.opentelemetry.io/collector/consumer/consumererror go.opentelemetry.io/collector/pdata/pmetric/pmetricotlp github.com/grafana/mimir/pkg/frontend/v1/frontendv1pb github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/prometheusremotewrite github.com/grafana/mimir/pkg/scheduler/queue github.com/grafana/mimir/pkg/frontend/v2/frontendv2pb github.com/grafana/mimir/pkg/util/servicediscovery github.com/grafana/mimir/pkg/util/push github.com/grafana/mimir/pkg/frontend/v1 github.com/grafana/mimir/pkg/scheduler/schedulerpb github.com/grafana/mimir/pkg/scheduler/schedulerdiscovery github.com/grafana/mimir/pkg/storage/chunk github.com/grafana/mimir/pkg/distributor github.com/grafana/mimir/pkg/querier/batch github.com/edsrzf/mmap-go github.com/grafana/mimir/pkg/util/activitytracker github.com/go-logr/logr github.com/grafana/mimir/pkg/frontend/v2 github.com/go-logr/logr/funcr go.opentelemetry.io/otel/internal/attribute go.opentelemetry.io/otel/attribute github.com/go-logr/stdr go.opentelemetry.io/otel/codes go.opentelemetry.io/otel/internal/baggage github.com/prometheus/prometheus/util/teststorage go.opentelemetry.io/otel/baggage go.opentelemetry.io/otel/trace github.com/grafana/mimir/pkg/querier/iterators github.com/grafana/mimir/pkg/storage/lazyquery github.com/grafana/mimir/pkg/storage/series go.opentelemetry.io/otel/propagation github.com/grafana/mimir/pkg/storegateway/storepb github.com/dustin/go-humanize github.com/grafana/mimir/pkg/util/pool go.opentelemetry.io/otel/internal/global github.com/grafana/mimir/pkg/util/chunkcompat go.opentelemetry.io/otel github.com/grafana/mimir/pkg/util/listblocks github.com/prometheus/prometheus/notifier github.com/prometheus/prometheus/util/stats github.com/prometheus/prometheus/util/gate github.com/prometheus/prometheus/util/logging github.com/prometheus/prometheus/promql go.opentelemetry.io/otel/metric/instrument/asyncfloat64 go.opentelemetry.io/otel/metric/instrument/asyncint64 go.opentelemetry.io/otel/metric/instrument/syncfloat64 go.opentelemetry.io/otel/metric/instrument/syncint64 go.opentelemetry.io/otel/semconv/internal go.opentelemetry.io/otel/metric go.opentelemetry.io/otel/metric/internal/global go.opentelemetry.io/otel/semconv/v1.12.0 github.com/grafana/mimir/pkg/storegateway/hintspb github.com/grafana/mimir/pkg/storegateway/storegatewaypb go.opentelemetry.io/otel/metric/global github.com/grafana/mimir/pkg/util/version go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp github.com/grafana/mimir/pkg/storegateway github.com/prometheus/client_golang/api github.com/prometheus/client_golang/api/prometheus/v1 github.com/prometheus/prometheus/storage/remote github.com/prometheus/prometheus/discovery/refresh github.com/spf13/afero/mem github.com/grafana/mimir/pkg/querier/engine github.com/prometheus/prometheus/template github.com/spf13/afero github.com/prometheus/prometheus/model/rulefmt github.com/prometheus/prometheus/rules github.com/grafana/mimir/pkg/ruler/rulespb github.com/grafana/mimir/pkg/scheduler github.com/prometheus/prometheus/util/httputil github.com/grafana/mimir/pkg/util/gziphandler github.com/grafana/mimir/pkg/ruler/rulestore/local github.com/prometheus/prometheus/web/api/v1 github.com/grafana/mimir/pkg/ruler/rulestore github.com/grafana/mimir/pkg/util/validation/exporter github.com/grafana/mimir/pkg/querier github.com/grafana/mimir/pkg/ruler/rulestore/bucketclient github.com/grafana/mimir/pkg/api/error github.com/grafana/mimir/pkg/frontend/querymiddleware/astmapper github.com/grafana/mimir/pkg/frontend/transport github.com/grafana/mimir/pkg/util/instrumentation github.com/grafana/mimir/pkg/frontend/querymiddleware github.com/grafana/mimir/pkg/usagestats github.com/grafana/mimir/pkg/querier/worker github.com/grafana/mimir/pkg/ingester github.com/grafana/mimir/pkg/util/fs github.com/grafana/mimir/pkg/util/noauth github.com/grafana/mimir/pkg/util/process github.com/grafana/mimir/pkg/ruler github.com/grafana/mimir/pkg/querier/tenantfederation github.com/grafana/mimir/pkg/util/fieldcategory github.com/grafana/mimir/pkg/util/usage github.com/grafana/mimir/pkg/continuoustest github.com/grafana/mimir/pkg/frontend database/sql github.com/rainycape/unidecode github.com/gosimple/slug github.com/grafana/mimir/pkg/flusher github.com/grafana-tools/sdk github.com/grafana/mimir/pkg/api github.com/grafana/mimir/pkg/mimirtool/minisdk github.com/grafana/mimir/pkg/mimirtool/rules/rwrulefmt github.com/grafana/mimir/pkg/mimirtool/analyze github.com/prometheus/client_golang/prometheus/collectors github.com/grafana/mimir/pkg/mimirtool/backfill github.com/grafana/mimir/cmd/mimir-continuous-test github.com/grafana/mimir/pkg/mimirtool/client github.com/dlclark/regexp2/syntax github.com/mitchellh/colorstring github.com/grafana/mimir/pkg/mimirtool/rules github.com/grafana/mimir/pkg/mimir github.com/alecthomas/template/parse github.com/dlclark/regexp2 github.com/alecthomas/template github.com/alecthomas/chroma go/scanner github.com/grafana/mimir/cmd/mimir github.com/grafana/mimir/tools/doc-generator/parse github.com/alecthomas/chroma/formatters/html github.com/grafana/mimir/pkg/mimirtool/config github.com/alecthomas/chroma/formatters/svg github.com/alecthomas/chroma/formatters github.com/alecthomas/chroma/lexers/internal github.com/alecthomas/chroma/styles github.com/alecthomas/chroma/lexers/a github.com/alecthomas/chroma/lexers/b github.com/alecthomas/chroma/lexers/p github.com/alecthomas/chroma/lexers/j github.com/alecthomas/chroma/lexers/c github.com/alecthomas/chroma/lexers/d github.com/alecthomas/chroma/lexers/e github.com/alecthomas/chroma/lexers/f github.com/alecthomas/chroma/lexers/i github.com/alecthomas/chroma/lexers/k github.com/alecthomas/chroma/lexers/l github.com/alecthomas/chroma/lexers/n github.com/alecthomas/chroma/lexers/o github.com/alecthomas/chroma/lexers/h github.com/alecthomas/chroma/lexers/q github.com/alecthomas/chroma/lexers/r github.com/alecthomas/chroma/lexers/t github.com/alecthomas/chroma/lexers/v github.com/alecthomas/chroma/lexers/w github.com/alecthomas/chroma/lexers/x github.com/alecthomas/chroma/lexers/y github.com/alecthomas/chroma/lexers/z github.com/alecthomas/chroma/lexers/circular github.com/alecthomas/chroma/lexers/g github.com/alecthomas/chroma/lexers/m github.com/alecthomas/chroma/lexers/s go/ast go/doc/comment internal/lazyregexp github.com/google/go-querystring/query golang.org/x/crypto/openpgp/errors golang.org/x/crypto/openpgp/armor go/doc compress/bzip2 golang.org/x/crypto/cast5 golang.org/x/crypto/openpgp/elgamal golang.org/x/crypto/openpgp/s2k image gopkg.in/alecthomas/kingpin.v2 github.com/alecthomas/chroma/lexers github.com/alecthomas/chroma/quick github.com/grafana/mimir/pkg/mimirtool/printer image/internal/imageutil github.com/grafana/mimir/tools/querytee image/jpeg golang.org/x/crypto/openpgp/packet github.com/grafana/mimir/cmd/query-tee github.com/grafana/mimir/pkg/mimirtool/commands golang.org/x/crypto/openpgp github.com/google/go-github/v32/github github.com/grafana/mimir/pkg/mimirtool/version github.com/grafana/mimir/cmd/mimirtool ==> Starting check()... === RUN TestConvertTenantBlocks --- PASS: TestConvertTenantBlocks (0.01s) === RUN TestConvertTenantBlocksDryMode --- PASS: TestConvertTenantBlocksDryMode (0.03s) PASS ok github.com/grafana/mimir/cmd/metaconvert 0.321s === RUN TestFlagParsing === RUN TestFlagParsing/common_yaml_inheritance_with_common_config_in_the_last_place === RUN TestFlagParsing/common_yaml_sets_a_value_but_specific_config_reverts_it_back === RUN TestFlagParsing/help-all === RUN TestFlagParsing/root_level_configuration_option_specified_as_an_empty_node_in_YAML_does_not_set_entire_config_to_zero_value === RUN TestFlagParsing/version === RUN TestFlagParsing/common_flag_inheritance === RUN TestFlagParsing/default_values === RUN TestFlagParsing/help === RUN TestFlagParsing/new_flag,_with_config === RUN TestFlagParsing/config_with_arguments_override === RUN TestFlagParsing/common_yaml_overridden_by_a_common_flag_and_specific_flag === RUN TestFlagParsing/unknown_flag === RUN TestFlagParsing/config_with_expand-env === RUN TestFlagParsing/user_visible_module_listing === RUN TestFlagParsing/user_visible_module_listing_flag_take_precedence_over_target_flag === RUN TestFlagParsing/common_yaml_inheritance_with_common_config_in_the_first_place === RUN TestFlagParsing/config === RUN TestFlagParsing/common_yaml_unmarshaling_is_strict === RUN TestFlagParsing/help-short --- PASS: TestFlagParsing (2.71s) --- PASS: TestFlagParsing/common_yaml_inheritance_with_common_config_in_the_last_place (0.32s) --- PASS: TestFlagParsing/common_yaml_sets_a_value_but_specific_config_reverts_it_back (0.20s) --- PASS: TestFlagParsing/help-all (0.11s) --- PASS: TestFlagParsing/root_level_configuration_option_specified_as_an_empty_node_in_YAML_does_not_set_entire_config_to_zero_value (0.21s) --- PASS: TestFlagParsing/version (0.04s) --- PASS: TestFlagParsing/common_flag_inheritance (0.16s) --- PASS: TestFlagParsing/default_values (0.16s) --- PASS: TestFlagParsing/help (0.09s) --- PASS: TestFlagParsing/new_flag,_with_config (0.08s) --- PASS: TestFlagParsing/config_with_arguments_override (0.10s) --- PASS: TestFlagParsing/common_yaml_overridden_by_a_common_flag_and_specific_flag (0.15s) --- PASS: TestFlagParsing/unknown_flag (0.13s) --- PASS: TestFlagParsing/config_with_expand-env (0.10s) --- PASS: TestFlagParsing/user_visible_module_listing (0.10s) --- PASS: TestFlagParsing/user_visible_module_listing_flag_take_precedence_over_target_flag (0.09s) --- PASS: TestFlagParsing/common_yaml_inheritance_with_common_config_in_the_first_place (0.27s) --- PASS: TestFlagParsing/config (0.20s) --- PASS: TestFlagParsing/common_yaml_unmarshaling_is_strict (0.09s) --- PASS: TestFlagParsing/help-short (0.09s) === RUN TestHelp === RUN TestHelp/basic === RUN TestHelp/all --- PASS: TestHelp (0.28s) --- PASS: TestHelp/basic (0.09s) --- PASS: TestHelp/all (0.18s) === RUN TestExpandEnvironmentVariables === RUN TestExpandEnvironmentVariables/x$y === RUN TestExpandEnvironmentVariables/x${y} === RUN TestExpandEnvironmentVariables/x$Y === RUN TestExpandEnvironmentVariables/x${Y} === RUN TestExpandEnvironmentVariables/x${Z:D} === RUN TestExpandEnvironmentVariables/x${Z:A_B_C_D} === RUN TestExpandEnvironmentVariables/x${Z:} === RUN TestExpandEnvironmentVariables/x$y:D --- PASS: TestExpandEnvironmentVariables (0.00s) --- PASS: TestExpandEnvironmentVariables/x$y (0.00s) --- PASS: TestExpandEnvironmentVariables/x${y} (0.00s) --- PASS: TestExpandEnvironmentVariables/x$Y (0.00s) --- PASS: TestExpandEnvironmentVariables/x${Y} (0.00s) --- PASS: TestExpandEnvironmentVariables/x${Z:D} (0.00s) --- PASS: TestExpandEnvironmentVariables/x${Z:A_B_C_D} (0.00s) --- PASS: TestExpandEnvironmentVariables/x${Z:} (0.00s) --- PASS: TestExpandEnvironmentVariables/x$y:D (0.00s) === RUN TestParseConfigFileParameter === RUN TestParseConfigFileParameter/#00 === RUN TestParseConfigFileParameter/--foo === RUN TestParseConfigFileParameter/-f_-a === RUN TestParseConfigFileParameter/--config.file=foo === RUN TestParseConfigFileParameter/--config.file_foo === RUN TestParseConfigFileParameter/--config.file=foo_--config.expand-env === RUN TestParseConfigFileParameter/--config.expand-env_--config.file=foo === RUN TestParseConfigFileParameter/--opt1_--config.file=foo === RUN TestParseConfigFileParameter/--opt1_--config.file_foo === RUN TestParseConfigFileParameter/--opt1_--config.file=foo_--config.expand-env === RUN TestParseConfigFileParameter/--opt1_--config.expand-env_--config.file=foo === RUN TestParseConfigFileParameter/--config.file=foo_--opt1 === RUN TestParseConfigFileParameter/--config.file_foo_--opt1 === RUN TestParseConfigFileParameter/--config.file=foo_--config.expand-env_--opt1 === RUN TestParseConfigFileParameter/--config.expand-env_--config.file=foo_--opt1 === RUN TestParseConfigFileParameter/--config.file=foo_--opt1_--config.expand-env === RUN TestParseConfigFileParameter/--config.expand-env_--opt1_--config.file=foo --- PASS: TestParseConfigFileParameter (0.01s) --- PASS: TestParseConfigFileParameter/#00 (0.00s) --- PASS: TestParseConfigFileParameter/--foo (0.00s) --- PASS: TestParseConfigFileParameter/-f_-a (0.00s) --- PASS: TestParseConfigFileParameter/--config.file=foo (0.00s) --- PASS: TestParseConfigFileParameter/--config.file_foo (0.00s) --- PASS: TestParseConfigFileParameter/--config.file=foo_--config.expand-env (0.00s) --- PASS: TestParseConfigFileParameter/--config.expand-env_--config.file=foo (0.00s) --- PASS: TestParseConfigFileParameter/--opt1_--config.file=foo (0.00s) --- PASS: TestParseConfigFileParameter/--opt1_--config.file_foo (0.00s) --- PASS: TestParseConfigFileParameter/--opt1_--config.file=foo_--config.expand-env (0.00s) --- PASS: TestParseConfigFileParameter/--opt1_--config.expand-env_--config.file=foo (0.00s) --- PASS: TestParseConfigFileParameter/--config.file=foo_--opt1 (0.00s) --- PASS: TestParseConfigFileParameter/--config.file_foo_--opt1 (0.00s) --- PASS: TestParseConfigFileParameter/--config.file=foo_--config.expand-env_--opt1 (0.00s) --- PASS: TestParseConfigFileParameter/--config.expand-env_--config.file=foo_--opt1 (0.00s) --- PASS: TestParseConfigFileParameter/--config.file=foo_--opt1_--config.expand-env (0.00s) --- PASS: TestParseConfigFileParameter/--config.expand-env_--opt1_--config.file=foo (0.00s) === RUN TestFieldCategoryOverridesNotStale --- PASS: TestFieldCategoryOverridesNotStale (0.04s) PASS ok github.com/grafana/mimir/cmd/mimir 3.848s ? github.com/grafana/mimir/cmd/mimir-continuous-test [no test files] ? github.com/grafana/mimir/cmd/mimirtool [no test files] === RUN TestMimirReadRoutes --- PASS: TestMimirReadRoutes (0.00s) PASS ok github.com/grafana/mimir/cmd/query-tee 0.210s ? github.com/grafana/mimir/integration [no test files] ? github.com/grafana/mimir/integration/ca [no test files] ? github.com/grafana/mimir/integration/e2emimir [no test files] === RUN TestResolveConfigs === RUN TestResolveConfigs/ResolveConfigs_annotates_values_that_match_defaults === RUN TestResolveConfigs/ResolveConfigs_sets_configuration_parameters_from_cli_arguments === RUN TestResolveConfigs/ResolveConfigs_sets_configuration_parameters_from_mounted_ConfigMaps === RUN TestResolveConfigs/ResolveConfigs_sets_configuration_parameters_from_mounted_Secrets --- PASS: TestResolveConfigs (1.42s) --- PASS: TestResolveConfigs/ResolveConfigs_annotates_values_that_match_defaults (0.34s) --- PASS: TestResolveConfigs/ResolveConfigs_sets_configuration_parameters_from_cli_arguments (0.34s) --- PASS: TestResolveConfigs/ResolveConfigs_sets_configuration_parameters_from_mounted_ConfigMaps (0.42s) --- PASS: TestResolveConfigs/ResolveConfigs_sets_configuration_parameters_from_mounted_Secrets (0.32s) PASS ok github.com/grafana/mimir/operations/compare-helm-with-jsonnet/plugins/resolve-config 1.970s === RUN TestMultitenantAlertmanager_GetStatusHandler --- PASS: TestMultitenantAlertmanager_GetStatusHandler (0.29s) === RUN TestAlertmanagerMetricsStore --- PASS: TestAlertmanagerMetricsStore (0.05s) === RUN TestAlertmanagerMetricsRemoval --- PASS: TestAlertmanagerMetricsRemoval (0.11s) === RUN TestIsHealthyForAlertmanagerOperations === PAUSE TestIsHealthyForAlertmanagerOperations === RUN Test_withCustomFunctions === RUN Test_withCustomFunctions/template_tenant_ID === RUN Test_withCustomFunctions/generate_grafana_explore_URL === RUN Test_withCustomFunctions/invalid_params_for_grafanaExploreURL --- PASS: Test_withCustomFunctions (0.03s) --- PASS: Test_withCustomFunctions/template_tenant_ID (0.00s) --- PASS: Test_withCustomFunctions/generate_grafana_explore_URL (0.00s) --- PASS: Test_withCustomFunctions/invalid_params_for_grafanaExploreURL (0.00s) === RUN TestDispatcherGroupLimits === RUN TestDispatcherGroupLimits/no_limit === RUN TestDispatcherGroupLimits/high_limit === RUN TestDispatcherGroupLimits/low_limit --- PASS: TestDispatcherGroupLimits (0.72s) --- PASS: TestDispatcherGroupLimits/no_limit (0.57s) --- PASS: TestDispatcherGroupLimits/high_limit (0.06s) --- PASS: TestDispatcherGroupLimits/low_limit (0.09s) === RUN TestDispatcherLoggerInsightKey --- PASS: TestDispatcherLoggerInsightKey (0.11s) === RUN TestAlertsLimiterWithNoLimits --- PASS: TestAlertsLimiterWithNoLimits (0.00s) === RUN TestAlertsLimiterWithCountLimit --- PASS: TestAlertsLimiterWithCountLimit (0.00s) === RUN TestAlertsLimiterWithSizeLimit --- PASS: TestAlertsLimiterWithSizeLimit (0.00s) === RUN TestAlertsLimiterWithSizeLimitAndAnnotationUpdate --- PASS: TestAlertsLimiterWithSizeLimitAndAnnotationUpdate (0.00s) === RUN TestAMConfigValidationAPI === RUN TestAMConfigValidationAPI/Should_return_error_if_the_alertmanager_config_contains_no_receivers === RUN TestAMConfigValidationAPI/Should_pass_if_the_alertmanager_config_is_valid === RUN TestAMConfigValidationAPI/Should_return_error_if_the_config_is_empty_due_to_wrong_indentation === RUN TestAMConfigValidationAPI/Should_return_error_if_the_alertmanager_config_is_empty_due_to_wrong_key === RUN TestAMConfigValidationAPI/Should_return_error_if_the_external_template_file_name_contains_an_absolute_path === RUN TestAMConfigValidationAPI/Should_return_error_if_the_external_template_file_name_contains_a_relative_path === RUN TestAMConfigValidationAPI/Should_return_error_if_the_external_template_file_name_is_not_a_valid_filename === RUN TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_the_root_/ === RUN TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_the_root_with_repeated_separators_/// === RUN TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_an_absolute_path === RUN TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_a_relative_path === RUN TestAMConfigValidationAPI/Should_pass_if_the_referenced_template_is_valid_filename === RUN TestAMConfigValidationAPI/Should_return_error_if_global_HTTP_password_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_HTTP_bearer_token_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_HTTP_credentials_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_OAuth2_client_secret_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_OAuth2_proxy_url_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_OAuth2_TLS_key_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_receiver's_HTTP_password_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_receiver's_HTTP_bearer_token_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_receiver's_HTTP_credentials_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_receiver's_OAuth2_client_secret_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_receiver's_OAuth2_proxy_url_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_slack_api_url_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_Slack_api_url_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_opsgenie_api_key_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_OpsGenie_api_key_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_victorops_api_key_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_VictorOps_api_key_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_PagerDuty_service_key_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_PagerDuty_routing_key_file_is_set === RUN TestAMConfigValidationAPI/should_return_error_if_template_is_wrong === RUN TestAMConfigValidationAPI/config_too_big === RUN TestAMConfigValidationAPI/config_size_OK === RUN TestAMConfigValidationAPI/templates_limit_reached === RUN TestAMConfigValidationAPI/templates_limit_not_reached === RUN TestAMConfigValidationAPI/template_size_limit_reached === RUN TestAMConfigValidationAPI/template_size_limit_ok === RUN TestAMConfigValidationAPI/Should_pass_if_template_uses_the_tenantID_custom_function === RUN TestAMConfigValidationAPI/Should_pass_if_template_uses_the_grafanaExploreURL_custom_function --- PASS: TestAMConfigValidationAPI (0.42s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_alertmanager_config_contains_no_receivers (0.00s) --- PASS: TestAMConfigValidationAPI/Should_pass_if_the_alertmanager_config_is_valid (0.04s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_config_is_empty_due_to_wrong_indentation (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_alertmanager_config_is_empty_due_to_wrong_key (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_external_template_file_name_contains_an_absolute_path (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_external_template_file_name_contains_a_relative_path (0.01s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_external_template_file_name_is_not_a_valid_filename (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_the_root_/ (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_the_root_with_repeated_separators_/// (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_an_absolute_path (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_a_relative_path (0.01s) --- PASS: TestAMConfigValidationAPI/Should_pass_if_the_referenced_template_is_valid_filename (0.04s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_HTTP_password_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_HTTP_bearer_token_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_HTTP_credentials_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_OAuth2_client_secret_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_OAuth2_proxy_url_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_OAuth2_TLS_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_receiver's_HTTP_password_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_receiver's_HTTP_bearer_token_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_receiver's_HTTP_credentials_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_receiver's_OAuth2_client_secret_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_receiver's_OAuth2_proxy_url_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_slack_api_url_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_Slack_api_url_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_opsgenie_api_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_OpsGenie_api_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_victorops_api_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_VictorOps_api_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_PagerDuty_service_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_PagerDuty_routing_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/should_return_error_if_template_is_wrong (0.04s) --- PASS: TestAMConfigValidationAPI/config_too_big (0.00s) --- PASS: TestAMConfigValidationAPI/config_size_OK (0.03s) --- PASS: TestAMConfigValidationAPI/templates_limit_reached (0.00s) --- PASS: TestAMConfigValidationAPI/templates_limit_not_reached (0.03s) --- PASS: TestAMConfigValidationAPI/template_size_limit_reached (0.00s) --- PASS: TestAMConfigValidationAPI/template_size_limit_ok (0.04s) --- PASS: TestAMConfigValidationAPI/Should_pass_if_template_uses_the_tenantID_custom_function (0.07s) --- PASS: TestAMConfigValidationAPI/Should_pass_if_template_uses_the_grafanaExploreURL_custom_function (0.04s) === RUN TestMultitenantAlertmanager_DeleteUserConfig --- PASS: TestMultitenantAlertmanager_DeleteUserConfig (0.00s) === RUN TestAMConfigListUserConfig --- PASS: TestAMConfigListUserConfig (0.55s) === RUN TestValidateAlertmanagerConfig === RUN TestValidateAlertmanagerConfig/*HTTPClientConfig === RUN TestValidateAlertmanagerConfig/HTTPClientConfig === RUN TestValidateAlertmanagerConfig/*TLSConfig === RUN TestValidateAlertmanagerConfig/*GlobalConfig.SMTPAuthPasswordFile === RUN TestValidateAlertmanagerConfig/GlobalConfig.SMTPAuthPasswordFile === RUN TestValidateAlertmanagerConfig/struct_containing_*HTTPClientConfig_as_direct_child === RUN TestValidateAlertmanagerConfig/struct_containing_*HTTPClientConfig_as_nested_child === RUN TestValidateAlertmanagerConfig/map_containing_*HTTPClientConfig === RUN TestValidateAlertmanagerConfig/map_containing_TLSConfig_as_nested_child === RUN TestValidateAlertmanagerConfig/TLSConfig === RUN TestValidateAlertmanagerConfig/*EmailConfig.AuthPasswordFile === RUN TestValidateAlertmanagerConfig/EmailConfig.AuthPasswordFile === RUN TestValidateAlertmanagerConfig/struct_containing_*HTTPClientConfig_as_nested_child_within_a_slice --- PASS: TestValidateAlertmanagerConfig (0.03s) --- PASS: TestValidateAlertmanagerConfig/*HTTPClientConfig (0.00s) --- PASS: TestValidateAlertmanagerConfig/HTTPClientConfig (0.00s) --- PASS: TestValidateAlertmanagerConfig/*TLSConfig (0.00s) --- PASS: TestValidateAlertmanagerConfig/*GlobalConfig.SMTPAuthPasswordFile (0.00s) --- PASS: TestValidateAlertmanagerConfig/GlobalConfig.SMTPAuthPasswordFile (0.00s) --- PASS: TestValidateAlertmanagerConfig/struct_containing_*HTTPClientConfig_as_direct_child (0.00s) --- PASS: TestValidateAlertmanagerConfig/struct_containing_*HTTPClientConfig_as_nested_child (0.00s) --- PASS: TestValidateAlertmanagerConfig/map_containing_*HTTPClientConfig (0.00s) --- PASS: TestValidateAlertmanagerConfig/map_containing_TLSConfig_as_nested_child (0.00s) --- PASS: TestValidateAlertmanagerConfig/TLSConfig (0.00s) --- PASS: TestValidateAlertmanagerConfig/*EmailConfig.AuthPasswordFile (0.00s) --- PASS: TestValidateAlertmanagerConfig/EmailConfig.AuthPasswordFile (0.00s) --- PASS: TestValidateAlertmanagerConfig/struct_containing_*HTTPClientConfig_as_nested_child_within_a_slice (0.00s) === RUN TestDistributor_DistributeRequest === RUN TestDistributor_DistributeRequest/Write_/alerts,_Simple_AM_request,_all_AM_healthy === RUN TestDistributor_DistributeRequest/Write_/alerts,_no_healthy_Alertmanagers === RUN TestDistributor_DistributeRequest/Write_/alerts,_1_healthy_Alertmanager_out_of_3 === RUN TestDistributor_DistributeRequest/Write_/alerts,_3_healthy_Alertmanagers_out_of_5 === RUN TestDistributor_DistributeRequest/Read_/v1/alerts_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Read_/v2/alerts_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Read_/v2/alerts/groups_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Write_/alerts/groups_not_supported === RUN TestDistributor_DistributeRequest/Read_/v1/silences_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Read_/v2/silences_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Write_/silences_is_sent_to_only_1_AM === RUN TestDistributor_DistributeRequest/Read_/v1/silence/id_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Read_/v2/silence/id_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Write_/silence/id_not_supported === RUN TestDistributor_DistributeRequest/Delete_/silence/id_is_sent_to_only_1_AM === RUN TestDistributor_DistributeRequest/Read_/status_is_sent_to_only_1_AM === RUN TestDistributor_DistributeRequest/Write_/status_not_supported === RUN TestDistributor_DistributeRequest/Read_/some/place/arbitrary_is_sent_to_only_1_AM === RUN TestDistributor_DistributeRequest/Read_/receivers_is_sent_to_only_1_AM === RUN TestDistributor_DistributeRequest/Write_/receivers_not_supported --- PASS: TestDistributor_DistributeRequest (0.11s) --- PASS: TestDistributor_DistributeRequest/Write_/alerts,_Simple_AM_request,_all_AM_healthy (0.00s) --- PASS: TestDistributor_DistributeRequest/Write_/alerts,_no_healthy_Alertmanagers (0.00s) --- PASS: TestDistributor_DistributeRequest/Write_/alerts,_1_healthy_Alertmanager_out_of_3 (0.01s) --- PASS: TestDistributor_DistributeRequest/Write_/alerts,_3_healthy_Alertmanagers_out_of_5 (0.01s) --- PASS: TestDistributor_DistributeRequest/Read_/v1/alerts_is_sent_to_3_AMs (0.02s) --- PASS: TestDistributor_DistributeRequest/Read_/v2/alerts_is_sent_to_3_AMs (0.01s) --- PASS: TestDistributor_DistributeRequest/Read_/v2/alerts/groups_is_sent_to_3_AMs (0.00s) --- PASS: TestDistributor_DistributeRequest/Write_/alerts/groups_not_supported (0.00s) --- PASS: TestDistributor_DistributeRequest/Read_/v1/silences_is_sent_to_3_AMs (0.00s) --- PASS: TestDistributor_DistributeRequest/Read_/v2/silences_is_sent_to_3_AMs (0.00s) --- PASS: TestDistributor_DistributeRequest/Write_/silences_is_sent_to_only_1_AM (0.00s) --- PASS: TestDistributor_DistributeRequest/Read_/v1/silence/id_is_sent_to_3_AMs (0.00s) --- PASS: TestDistributor_DistributeRequest/Read_/v2/silence/id_is_sent_to_3_AMs (0.00s) --- PASS: TestDistributor_DistributeRequest/Write_/silence/id_not_supported (0.00s) --- PASS: TestDistributor_DistributeRequest/Delete_/silence/id_is_sent_to_only_1_AM (0.00s) --- PASS: TestDistributor_DistributeRequest/Read_/status_is_sent_to_only_1_AM (0.00s) --- PASS: TestDistributor_DistributeRequest/Write_/status_not_supported (0.00s) --- PASS: TestDistributor_DistributeRequest/Read_/some/place/arbitrary_is_sent_to_only_1_AM (0.00s) --- PASS: TestDistributor_DistributeRequest/Read_/receivers_is_sent_to_only_1_AM (0.00s) --- PASS: TestDistributor_DistributeRequest/Write_/receivers_not_supported (0.00s) === RUN TestMultitenantAlertmanagerConfig_Validate === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_with_empty_external_URL === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_external_URL_ends_with_/ === RUN TestMultitenantAlertmanagerConfig_Validate/should_succeed_if_external_URL_does_not_end_with_/ === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_external_URL_has_no_hostname === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_zone_aware_is_enabled_but_zone_is_not_set === RUN TestMultitenantAlertmanagerConfig_Validate/should_pass_if_the_URL_just_contains_the_path_with_the_leading_/ === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_the_URL_just_contains_the_hostname_(because_it_can't_be_disambiguated_with_a_path) === RUN TestMultitenantAlertmanagerConfig_Validate/should_pass_with_default_config === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_persistent_interval_is_0 === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_persistent_interval_is_negative === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_external_URL_has_no_scheme --- PASS: TestMultitenantAlertmanagerConfig_Validate (0.03s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_with_empty_external_URL (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_external_URL_ends_with_/ (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_succeed_if_external_URL_does_not_end_with_/ (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_external_URL_has_no_hostname (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_zone_aware_is_enabled_but_zone_is_not_set (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_pass_if_the_URL_just_contains_the_path_with_the_leading_/ (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_the_URL_just_contains_the_hostname_(because_it_can't_be_disambiguated_with_a_path) (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_pass_with_default_config (0.01s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_persistent_interval_is_0 (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_persistent_interval_is_negative (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_external_URL_has_no_scheme (0.00s) === RUN TestMultitenantAlertmanager_relativeDataDir --- PASS: TestMultitenantAlertmanager_relativeDataDir (0.25s) === RUN TestMultitenantAlertmanager_loadAndSyncConfigs --- PASS: TestMultitenantAlertmanager_loadAndSyncConfigs (0.60s) === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webex_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webex_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webex_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webex_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=false === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=true === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=false === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=false === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webex_firewall_enabled=false === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=false multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=sns msg="SNS message successfully published" message_id=null sequencenumber=null level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=sns integration=sns[0] msg="Notify success" attempts=1 === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=false === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webex_firewall_enabled=false multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="ring doesn't exist in KV store yet" level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=webex incident={}:{} level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=webex integration=webex[0] msg="Notify success" attempts=1 === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=true === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=false multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="ring doesn't exist in KV store yet" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=pagerduty incident={}:{} eventType=trigger level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=pagerduty integration=pagerduty[0] msg="Notify success" attempts=1 === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=false === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=false multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=discord incident={}:{} level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=discord integration=discord[0] msg="Notify success" attempts=1 === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=true === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=false multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=slack integration=slack[0] msg="Notify success" attempts=1 === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=false multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=wechat incident={}:{} level=error component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="wechat/wechat[0]: notify retry canceled due to unrecoverable error after 1 attempts: EOF" === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=true === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=true multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=opsgenie alert={}:{} level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=opsgenie integration=opsgenie[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"http://127.0.0.1:37169/v2/alerts\": dial tcp 127.0.0.1:37169: blocked address" level=debug component=MultiTenantAlertmanager user=user-1 integration=opsgenie alert={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=opsgenie alert={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=opsgenie alert={}:{} === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=true === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=true multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=discord incident={}:{} level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=discord integration=discord[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"\": dial tcp 127.0.0.1:43933: blocked address" level=debug component=MultiTenantAlertmanager user=user-1 integration=discord incident={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=discord incident={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=discord incident={}:{} === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=false === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=true multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=slack integration=slack[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"\": dial tcp 127.0.0.1:46351: blocked address" === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webex_firewall_enabled=true === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=false multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=telegram integration=telegram[0] msg="Notify attempt failed, will retry later" attempts=1 err="telebot: unexpected end of JSON input" === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=false multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=webhook integration=webhook[0] msg="Notify success" attempts=1 === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=true === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=true multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=sns integration=sns[0] msg="Notify attempt failed, will retry later" attempts=1 err="RequestError: send request failed\ncaused by: Post \"http://127.0.0.1:37193/\": dial tcp 127.0.0.1:37193: blocked address" === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=false multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="ring doesn't exist in KV store yet" level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=opsgenie alert={}:{} level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=opsgenie integration=opsgenie[0] msg="Notify success" attempts=1 === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=true === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=true multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="ring doesn't exist in KV store yet" level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=wechat incident={}:{} level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=wechat integration=wechat[0] msg="Notify attempt failed, will retry later" attempts=1 err="Get \"\": dial tcp 127.0.0.1:46551: blocked address" level=debug component=MultiTenantAlertmanager user=user-1 integration=wechat incident={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=wechat incident={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=wechat incident={}:{} === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=true === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webex_firewall_enabled=true multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="ring doesn't exist in KV store yet" level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=webex incident={}:{} level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=webex integration=webex[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"\": dial tcp 127.0.0.1:34775: blocked address" level=debug component=MultiTenantAlertmanager user=user-1 integration=webex incident={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=webex incident={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=webex incident={}:{} === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=false multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=true === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=true multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=pagerduty incident={}:{} eventType=trigger level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=pagerduty integration=pagerduty[0] msg="Notify attempt failed, will retry later" attempts=1 err="failed to post message to PagerDuty: Post \"http://127.0.0.1:40599\": dial tcp 127.0.0.1:40599: blocked address" level=debug component=MultiTenantAlertmanager user=user-1 integration=pagerduty incident={}:{} eventType=trigger level=debug component=MultiTenantAlertmanager user=user-1 integration=pagerduty incident={}:{} eventType=trigger level=debug component=MultiTenantAlertmanager user=user-1 integration=pagerduty incident={}:{} eventType=trigger === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=true multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=telegram integration=telegram[0] msg="Notify attempt failed, will retry later" attempts=1 err="telebot: Post \"http://127.0.0.1:39813/botxxx/sendMessage\": dial tcp 127.0.0.1:39813: blocked address" === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=true multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="ring doesn't exist in KV store yet" level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=webhook integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"http://127.0.0.1:42581\": dial tcp 127.0.0.1:42581: blocked address" === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=true multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="ring doesn't exist in KV store yet" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=webhook integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://www.google.com\": proxyconnect tcp: dial tcp 127.0.0.1:43253: blocked address" --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled (0.00s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=false (0.26s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webex_firewall_enabled=false (0.44s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=false (0.52s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=false (0.36s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=false (0.34s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=false (0.45s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=true (3.38s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=true (3.36s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=true (3.25s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=false (0.31s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=false (0.34s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=true (3.30s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=false (0.52s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=true (3.36s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webex_firewall_enabled=true (3.40s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=false (0.29s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=true (3.35s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=true (3.29s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=true (3.41s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=true (3.30s) === RUN TestMultitenantAlertmanager_deleteUnusedLocalUserState --- PASS: TestMultitenantAlertmanager_deleteUnusedLocalUserState (0.49s) === RUN TestMultitenantAlertmanager_zoneAwareSharding level=info component=MultiTenantAlertmanager msg="ring doesn't exist in KV store yet" level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=instance-1 ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=instance-2 ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=instance-3 ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=3 level=debug component=MultiTenantAlertmanager msg="setting config" user=user3 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user3 level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="Waiting for notification and silences to settle..." level=debug component=MultiTenantAlertmanager msg="contacting replica for full state" user=user3 addr=127.0.0.1-3:0 level=debug component=MultiTenantAlertmanager msg="setting config" user=user1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user1 level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="Waiting for notification and silences to settle..." level=debug component=MultiTenantAlertmanager msg="contacting replica for full state" user=user1 addr=127.0.0.1-3:0 level=debug component=MultiTenantAlertmanager msg="setting config" user=user2 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user2 level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="Waiting for notification and silences to settle..." level=debug component=MultiTenantAlertmanager msg="contacting replica for full state" user=user2 addr=127.0.0.1-3:0 level=error component=MultiTenantAlertmanager msg="rpc reading state from replica failed" addr=127.0.0.1-3:0 user=user2 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup 127.0.0.1-3: no such host\"" level=error component=MultiTenantAlertmanager msg="rpc reading state from replica failed" addr=127.0.0.1-3:0 user=user3 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup 127.0.0.1-3: no such host\"" level=error component=MultiTenantAlertmanager msg="rpc reading state from replica failed" addr=127.0.0.1-3:0 user=user1 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup 127.0.0.1-3: no such host\"" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="unable to read state from other Alertmanager replicas; trying to read from storage" err="failed to read state from any replica" level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="unable to read state from other Alertmanager replicas; trying to read from storage" err="failed to read state from any replica" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="unable to read state from other Alertmanager replicas; trying to read from storage" err="failed to read state from any replica" level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=3 level=debug component=MultiTenantAlertmanager msg="setting config" user=user2 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user2 level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="Waiting for notification and silences to settle..." level=debug component=MultiTenantAlertmanager msg="contacting replica for full state" user=user2 addr=127.0.0.1-1:0 level=debug component=MultiTenantAlertmanager msg="setting config" user=user3 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user3 level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="Waiting for notification and silences to settle..." level=debug component=MultiTenantAlertmanager msg="contacting replica for full state" user=user3 addr=127.0.0.1-1:0 level=debug component=MultiTenantAlertmanager msg="setting config" user=user1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user1 level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="Waiting for notification and silences to settle..." level=debug component=MultiTenantAlertmanager msg="contacting replica for full state" user=user1 addr=127.0.0.1-1:0 level=error component=MultiTenantAlertmanager msg="rpc reading state from replica failed" addr=127.0.0.1-1:0 user=user2 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup 127.0.0.1-1: no such host\"" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="unable to read state from other Alertmanager replicas; trying to read from storage" err="failed to read state from any replica" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="no state for user in storage; proceeding" level=error component=MultiTenantAlertmanager msg="rpc reading state from replica failed" addr=127.0.0.1-1:0 user=user1 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup 127.0.0.1-1: no such host\"" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="unable to read state from other Alertmanager replicas; trying to read from storage" err="failed to read state from any replica" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="no state for user in storage; proceeding" level=error component=MultiTenantAlertmanager msg="rpc reading state from replica failed" addr=127.0.0.1-1:0 user=user3 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup 127.0.0.1-1: no such host\"" level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="unable to read state from other Alertmanager replicas; trying to read from storage" err="failed to read state from any replica" level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="ring lifecycler is shutting down" ring=alertmanager level=info component=MultiTenantAlertmanager msg="unregistering instance from ring" ring=alertmanager level=debug component=MultiTenantAlertmanager user=user2 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user2 component=nflog msg="Maintenance done" duration=633µs size=0 level=debug component=MultiTenantAlertmanager user=user2 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user2 component=silences msg="Maintenance done" duration=360µs size=0 level=info component=MultiTenantAlertmanager msg="instance removed from the ring" ring=alertmanager level=debug component=MultiTenantAlertmanager user=user3 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user3 component=nflog msg="Maintenance done" duration=398µs size=0 level=debug component=MultiTenantAlertmanager user=user3 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user3 component=silences msg="Maintenance done" duration=533µs size=0 level=debug component=MultiTenantAlertmanager user=user1 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user1 component=nflog msg="Maintenance done" duration=569.999µs size=0 level=debug component=MultiTenantAlertmanager user=user1 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user1 component=silences msg="Maintenance done" duration=343µs size=0 level=info component=MultiTenantAlertmanager msg="ring lifecycler is shutting down" ring=alertmanager level=info component=MultiTenantAlertmanager msg="unregistering instance from ring" ring=alertmanager level=info component=MultiTenantAlertmanager msg="instance removed from the ring" ring=alertmanager level=info component=MultiTenantAlertmanager msg="ring lifecycler is shutting down" ring=alertmanager level=info component=MultiTenantAlertmanager msg="unregistering instance from ring" ring=alertmanager level=debug component=MultiTenantAlertmanager user=user3 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user3 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user3 component=nflog msg="Maintenance done" duration=552µs size=0 level=debug component=MultiTenantAlertmanager user=user3 component=silences msg="Maintenance done" duration=531µs size=0 level=debug component=MultiTenantAlertmanager user=user1 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user1 component=silences msg="Running maintenance" level=info component=MultiTenantAlertmanager msg="instance removed from the ring" ring=alertmanager level=debug component=MultiTenantAlertmanager user=user1 component=nflog msg="Maintenance done" duration=310µs size=0 level=debug component=MultiTenantAlertmanager user=user1 component=silences msg="Maintenance done" duration=289µs size=0 level=debug component=MultiTenantAlertmanager user=user2 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user2 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user2 component=nflog msg="Maintenance done" duration=435µs size=0 level=debug component=MultiTenantAlertmanager user=user2 component=silences msg="Maintenance done" duration=363µs size=0 --- PASS: TestMultitenantAlertmanager_zoneAwareSharding (0.87s) === RUN TestMultitenantAlertmanager_deleteUnusedRemoteUserState level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=instance-1 ring=alertmanager level=info component=MultiTenantAlertmanager msg="ring doesn't exist in KV store yet" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=instance-2 ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user2 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user2 level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user1 level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="no state for user in storage; proceeding" level=debug component=MultiTenantAlertmanager user=user2 msg="persisting state" user=user2 level=debug component=MultiTenantAlertmanager user=user1 msg="persisting state" user=user1 level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user2 level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user1 level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user2 level=info component=MultiTenantAlertmanager msg="deleted remote state for user" user=user1 level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="deactivating per-tenant alertmanager" user=user1 level=debug component=MultiTenantAlertmanager user=user1 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user1 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user1 component=silences msg="Maintenance done" duration=491.999µs size=0 level=debug component=MultiTenantAlertmanager user=user1 component=nflog msg="Maintenance done" duration=1.777999ms size=0 level=info component=MultiTenantAlertmanager msg="deactivated per-tenant alertmanager" user=user1 level=info component=MultiTenantAlertmanager msg="deleted local directory for user" dir=/tmp/TestMultitenantAlertmanager_deleteUnusedRemoteUserState4015184535/002/user1 user=user1 level=info component=MultiTenantAlertmanager msg="ring lifecycler is shutting down" ring=alertmanager level=info component=MultiTenantAlertmanager msg="unregistering instance from ring" ring=alertmanager level=info component=MultiTenantAlertmanager msg="instance removed from the ring" ring=alertmanager level=info component=MultiTenantAlertmanager msg="ring lifecycler is shutting down" ring=alertmanager level=debug component=MultiTenantAlertmanager user=user2 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user2 component=silences msg="Running maintenance" level=info component=MultiTenantAlertmanager msg="unregistering instance from ring" ring=alertmanager level=info component=MultiTenantAlertmanager msg="instance removed from the ring" ring=alertmanager level=debug component=MultiTenantAlertmanager user=user2 component=silences msg="Maintenance done" duration=554µs size=0 level=debug component=MultiTenantAlertmanager user=user2 component=nflog msg="Maintenance done" duration=1.799ms size=0 --- PASS: TestMultitenantAlertmanager_deleteUnusedRemoteUserState (1.29s) === RUN TestMultitenantAlertmanager_ServeHTTP --- PASS: TestMultitenantAlertmanager_ServeHTTP (0.24s) === RUN TestMultitenantAlertmanager_ServeHTTPWithFallbackConfig --- PASS: TestMultitenantAlertmanager_ServeHTTPWithFallbackConfig (0.38s) === RUN TestMultitenantAlertmanager_ServeHTTPBeforeSyncFailsIfConfigExists --- PASS: TestMultitenantAlertmanager_ServeHTTPBeforeSyncFailsIfConfigExists (0.26s) === RUN TestMultitenantAlertmanager_InitialSync === RUN TestMultitenantAlertmanager_InitialSync/with_no_instance_in_the_ring === RUN TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_PENDING_state_and_no_tokens === RUN TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_JOINING_state_and_some_tokens === RUN TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_ACTIVE_state_and_all_tokens === RUN TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_LEAVING_state_and_all_tokens --- PASS: TestMultitenantAlertmanager_InitialSync (0.85s) --- PASS: TestMultitenantAlertmanager_InitialSync/with_no_instance_in_the_ring (0.33s) --- PASS: TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_PENDING_state_and_no_tokens (0.13s) --- PASS: TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_JOINING_state_and_some_tokens (0.16s) --- PASS: TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_ACTIVE_state_and_all_tokens (0.21s) --- PASS: TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_LEAVING_state_and_all_tokens (0.02s) === RUN TestMultitenantAlertmanager_PerTenantSharding === RUN TestMultitenantAlertmanager_PerTenantSharding/1_instance,_RF_=_1 === RUN TestMultitenantAlertmanager_PerTenantSharding/2_instances,_RF_=_1 === RUN TestMultitenantAlertmanager_PerTenantSharding/3_instances,_RF_=_2 === RUN TestMultitenantAlertmanager_PerTenantSharding/5_instances,_RF_=_3 --- PASS: TestMultitenantAlertmanager_PerTenantSharding (12.84s) --- PASS: TestMultitenantAlertmanager_PerTenantSharding/1_instance,_RF_=_1 (1.49s) --- PASS: TestMultitenantAlertmanager_PerTenantSharding/2_instances,_RF_=_1 (2.15s) --- PASS: TestMultitenantAlertmanager_PerTenantSharding/3_instances,_RF_=_2 (4.02s) --- PASS: TestMultitenantAlertmanager_PerTenantSharding/5_instances,_RF_=_3 (5.17s) === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/when_an_instance_is_added_to_the_ring === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/when_an_instance_is_removed_from_the_ring === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_sync_when_an_instance_changes_state === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_sync_when_an_healthy_instance_becomes_unhealthy === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_sync_when_an_unhealthy_instance_becomes_healthy === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_NOT_sync_when_an_instance_updates_the_heartbeat === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_NOT_sync_when_an_instance_is_auto-forgotten_in_the_ring_but_was_already_unhealthy_in_the_previous_state --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges (2.27s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/when_an_instance_is_added_to_the_ring (0.35s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/when_an_instance_is_removed_from_the_ring (0.44s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_sync_when_an_instance_changes_state (0.27s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_sync_when_an_healthy_instance_becomes_unhealthy (0.49s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_sync_when_an_unhealthy_instance_becomes_healthy (0.28s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_NOT_sync_when_an_instance_updates_the_heartbeat (0.18s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_NOT_sync_when_an_instance_is_auto-forgotten_in_the_ring_but_was_already_unhealthy_in_the_previous_state (0.26s) === RUN TestMultitenantAlertmanager_RingLifecyclerShouldAutoForgetUnhealthyInstances --- PASS: TestMultitenantAlertmanager_RingLifecyclerShouldAutoForgetUnhealthyInstances (0.31s) === RUN TestMultitenantAlertmanager_InitialSyncFailure --- PASS: TestMultitenantAlertmanager_InitialSyncFailure (0.13s) === RUN TestAlertmanager_ReplicasPosition --- PASS: TestAlertmanager_ReplicasPosition (1.01s) === RUN TestAlertmanager_StateReplication === RUN TestAlertmanager_StateReplication/RF_=_1,_1_instance === RUN TestAlertmanager_StateReplication/RF_=_2,_2_instances === RUN TestAlertmanager_StateReplication/RF_=_3,_10_instance --- PASS: TestAlertmanager_StateReplication (13.58s) --- PASS: TestAlertmanager_StateReplication/RF_=_1,_1_instance (1.42s) --- PASS: TestAlertmanager_StateReplication/RF_=_2,_2_instances (2.48s) --- PASS: TestAlertmanager_StateReplication/RF_=_3,_10_instance (9.68s) === RUN TestAlertmanager_StateReplication_InitialSyncFromPeers === RUN TestAlertmanager_StateReplication_InitialSyncFromPeers/RF_=_2 === RUN TestAlertmanager_StateReplication_InitialSyncFromPeers/RF_=_3 --- PASS: TestAlertmanager_StateReplication_InitialSyncFromPeers (2.18s) --- PASS: TestAlertmanager_StateReplication_InitialSyncFromPeers/RF_=_2 (0.87s) --- PASS: TestAlertmanager_StateReplication_InitialSyncFromPeers/RF_=_3 (1.31s) === RUN TestSafeTemplateFilepath === RUN TestSafeTemplateFilepath/template_name_starting_with_/ === RUN TestSafeTemplateFilepath/root_dir === RUN TestSafeTemplateFilepath/root_dir_2 === RUN TestSafeTemplateFilepath/should_fail_if_the_provided_template_is_escaping_the_dir === RUN TestSafeTemplateFilepath/escaping_template_name_that_has_prefix_of_dir_(tmp_is_prefix_of_tmpfile) === RUN TestSafeTemplateFilepath/empty_template_name === RUN TestSafeTemplateFilepath/dot_template_name === RUN TestSafeTemplateFilepath/should_succeed_if_the_provided_template_is_a_filename --- PASS: TestSafeTemplateFilepath (0.01s) --- PASS: TestSafeTemplateFilepath/template_name_starting_with_/ (0.00s) --- PASS: TestSafeTemplateFilepath/root_dir (0.00s) --- PASS: TestSafeTemplateFilepath/root_dir_2 (0.00s) --- PASS: TestSafeTemplateFilepath/should_fail_if_the_provided_template_is_escaping_the_dir (0.00s) --- PASS: TestSafeTemplateFilepath/escaping_template_name_that_has_prefix_of_dir_(tmp_is_prefix_of_tmpfile) (0.00s) --- PASS: TestSafeTemplateFilepath/empty_template_name (0.00s) --- PASS: TestSafeTemplateFilepath/dot_template_name (0.00s) --- PASS: TestSafeTemplateFilepath/should_succeed_if_the_provided_template_is_a_filename (0.01s) === RUN TestStoreTemplateFile --- PASS: TestStoreTemplateFile (0.00s) === RUN TestMultitenantAlertmanager_verifyRateLimitedEmailConfig --- PASS: TestMultitenantAlertmanager_verifyRateLimitedEmailConfig (0.26s) === RUN TestRateLimitedNotifier --- PASS: TestRateLimitedNotifier (1.00s) === RUN TestStatePersister_Position0ShouldWrite --- PASS: TestStatePersister_Position0ShouldWrite (6.10s) === RUN TestStatePersister_Position1ShouldNotWrite --- PASS: TestStatePersister_Position1ShouldNotWrite (5.00s) === RUN TestStateReplication === RUN TestStateReplication/with_a_replication_factor_of_<=_1,_state_is_not_replicated_but_loaded_from_storage. === RUN TestStateReplication/with_a_replication_factor_of_>_1,_state_is_broadcasted_for_replication. --- PASS: TestStateReplication (0.21s) --- PASS: TestStateReplication/with_a_replication_factor_of_<=_1,_state_is_not_replicated_but_loaded_from_storage. (0.10s) --- PASS: TestStateReplication/with_a_replication_factor_of_>_1,_state_is_broadcasted_for_replication. (0.11s) === RUN TestStateReplication_Settle === RUN TestStateReplication_Settle/with_a_replication_factor_of_<=_1,_no_state_can_be_read_from_peers. === RUN TestStateReplication_Settle/with_a_replication_factor_of_>_1,_state_is_read_from_all_peers. === RUN TestStateReplication_Settle/with_full_state_having_no_parts,_nothing_is_merged. === RUN TestStateReplication_Settle/with_an_unknown_key,_parts_in_the_same_state_are_merged. === RUN TestStateReplication_Settle/with_an_unknown_key,_parts_in_other_states_are_merged. === RUN TestStateReplication_Settle/when_reading_from_replicas_fails,_state_is_read_from_storage. === RUN TestStateReplication_Settle/when_reading_from_replicas_and_from_storage_fails,_still_become_ready. === RUN TestStateReplication_Settle/when_user_not_found_in_all_replicas_and_storage,_read_not_counted_as_failure_and_still_become_ready. === RUN TestStateReplication_Settle/when_reading_the_full_state_takes_too_long,_hit_timeout_but_become_ready. --- PASS: TestStateReplication_Settle (1.04s) --- PASS: TestStateReplication_Settle/with_a_replication_factor_of_<=_1,_no_state_can_be_read_from_peers. (0.00s) --- PASS: TestStateReplication_Settle/with_a_replication_factor_of_>_1,_state_is_read_from_all_peers. (0.00s) --- PASS: TestStateReplication_Settle/with_full_state_having_no_parts,_nothing_is_merged. (0.00s) --- PASS: TestStateReplication_Settle/with_an_unknown_key,_parts_in_the_same_state_are_merged. (0.00s) --- PASS: TestStateReplication_Settle/with_an_unknown_key,_parts_in_other_states_are_merged. (0.01s) --- PASS: TestStateReplication_Settle/when_reading_from_replicas_fails,_state_is_read_from_storage. (0.00s) --- PASS: TestStateReplication_Settle/when_reading_from_replicas_and_from_storage_fails,_still_become_ready. (0.00s) --- PASS: TestStateReplication_Settle/when_user_not_found_in_all_replicas_and_storage,_read_not_counted_as_failure_and_still_become_ready. (0.00s) --- PASS: TestStateReplication_Settle/when_reading_the_full_state_takes_too_long,_hit_timeout_but_become_ready. (1.00s) === RUN TestStateReplication_GetFullState === RUN TestStateReplication_GetFullState/no_keys === RUN TestStateReplication_GetFullState/zero_length_data === RUN TestStateReplication_GetFullState/keys_with_data --- PASS: TestStateReplication_GetFullState (0.00s) --- PASS: TestStateReplication_GetFullState/no_keys (0.00s) --- PASS: TestStateReplication_GetFullState/zero_length_data (0.00s) --- PASS: TestStateReplication_GetFullState/keys_with_data (0.00s) === CONT TestIsHealthyForAlertmanagerOperations === RUN TestIsHealthyForAlertmanagerOperations/ACTIVE_instance_with_last_keepalive_newer_than_timeout === RUN TestIsHealthyForAlertmanagerOperations/ACTIVE_instance_with_last_keepalive_older_than_timeout === RUN TestIsHealthyForAlertmanagerOperations/JOINING_instance_with_last_keepalive_newer_than_timeout === RUN TestIsHealthyForAlertmanagerOperations/LEAVING_instance_with_last_keepalive_newer_than_timeout === RUN TestIsHealthyForAlertmanagerOperations/PENDING_instance_with_last_keepalive_newer_than_timeout --- PASS: TestIsHealthyForAlertmanagerOperations (0.00s) --- PASS: TestIsHealthyForAlertmanagerOperations/ACTIVE_instance_with_last_keepalive_newer_than_timeout (0.00s) --- PASS: TestIsHealthyForAlertmanagerOperations/ACTIVE_instance_with_last_keepalive_older_than_timeout (0.00s) --- PASS: TestIsHealthyForAlertmanagerOperations/JOINING_instance_with_last_keepalive_newer_than_timeout (0.00s) --- PASS: TestIsHealthyForAlertmanagerOperations/LEAVING_instance_with_last_keepalive_newer_than_timeout (0.00s) --- PASS: TestIsHealthyForAlertmanagerOperations/PENDING_instance_with_last_keepalive_newer_than_timeout (0.00s) PASS ok github.com/grafana/mimir/pkg/alertmanager 65.142s ? github.com/grafana/mimir/pkg/alertmanager/alertmanagerpb [no test files] ? github.com/grafana/mimir/pkg/alertmanager/alertspb [no test files] === RUN TestAlertStore_ListAllUsers --- PASS: TestAlertStore_ListAllUsers (0.00s) === RUN TestAlertStore_SetAndGetAlertConfig --- PASS: TestAlertStore_SetAndGetAlertConfig (0.00s) === RUN TestStore_GetAlertConfigs --- PASS: TestStore_GetAlertConfigs (0.00s) === RUN TestAlertStore_DeleteAlertConfig --- PASS: TestAlertStore_DeleteAlertConfig (0.00s) === RUN TestBucketAlertStore_GetSetDeleteFullState --- PASS: TestBucketAlertStore_GetSetDeleteFullState (0.00s) PASS ok github.com/grafana/mimir/pkg/alertmanager/alertstore 0.332s testing: warning: no tests to run PASS ok github.com/grafana/mimir/pkg/alertmanager/alertstore/bucketclient 0.171s [no tests to run] === RUN TestStore_ListAllUsers --- PASS: TestStore_ListAllUsers (0.01s) === RUN TestStore_GetAlertConfig --- PASS: TestStore_GetAlertConfig (0.01s) === RUN TestStore_GetAlertConfigs --- PASS: TestStore_GetAlertConfigs (0.00s) === RUN TestStore_FullState --- PASS: TestStore_FullState (0.00s) PASS ok github.com/grafana/mimir/pkg/alertmanager/alertstore/local 0.168s === RUN TestV1Alerts --- PASS: TestV1Alerts (0.00s) === RUN TestMergeV1Alerts === RUN TestMergeV1Alerts/no_alerts,_should_return_an_empty_list === RUN TestMergeV1Alerts/one_alert,_should_return_the_alert === RUN TestMergeV1Alerts/two_alerts,_should_return_two_alerts === RUN TestMergeV1Alerts/three_alerts,_should_return_three_alerts === RUN TestMergeV1Alerts/three_alerts_out_of_order,_should_return_three_alerts_in_fingerprint_order === RUN TestMergeV1Alerts/two_identical_alerts,_should_return_one_alert === RUN TestMergeV1Alerts/two_identical_alerts_plus_another,_should_return_two_alerts === RUN TestMergeV1Alerts/two_duplicates_out_of_sync_alerts,_should_return_first_seen_alert === RUN TestMergeV1Alerts/two_duplicates_plus_others,_should_return_first_seen_alert_and_others --- PASS: TestMergeV1Alerts (0.00s) --- PASS: TestMergeV1Alerts/no_alerts,_should_return_an_empty_list (0.00s) --- PASS: TestMergeV1Alerts/one_alert,_should_return_the_alert (0.00s) --- PASS: TestMergeV1Alerts/two_alerts,_should_return_two_alerts (0.00s) --- PASS: TestMergeV1Alerts/three_alerts,_should_return_three_alerts (0.00s) --- PASS: TestMergeV1Alerts/three_alerts_out_of_order,_should_return_three_alerts_in_fingerprint_order (0.00s) --- PASS: TestMergeV1Alerts/two_identical_alerts,_should_return_one_alert (0.00s) --- PASS: TestMergeV1Alerts/two_identical_alerts_plus_another,_should_return_two_alerts (0.00s) --- PASS: TestMergeV1Alerts/two_duplicates_out_of_sync_alerts,_should_return_first_seen_alert (0.00s) --- PASS: TestMergeV1Alerts/two_duplicates_plus_others,_should_return_first_seen_alert_and_others (0.00s) === RUN TestV1SilenceID_ReturnsNewestSilence --- PASS: TestV1SilenceID_ReturnsNewestSilence (0.00s) === RUN TestV1SilenceID_InvalidDifferentIDs --- PASS: TestV1SilenceID_InvalidDifferentIDs (0.00s) === RUN TestV1Silences --- PASS: TestV1Silences (0.00s) === RUN TestV2AlertGroups --- PASS: TestV2AlertGroups (0.00s) === RUN TestMergeV2AlertGroups === RUN TestMergeV2AlertGroups/no_groups,_should_return_no_groups === RUN TestMergeV2AlertGroups/one_group_with_one_alert,_should_return_one_group === RUN TestMergeV2AlertGroups/two_groups_with_different_labels,_should_return_two_groups === RUN TestMergeV2AlertGroups/two_groups_with_different_receiver,_should_return_two_groups === RUN TestMergeV2AlertGroups/two_identical_groups_with_different_alerts,_should_return_one_group_with_two_alerts === RUN TestMergeV2AlertGroups/two_identical_groups_with_identical_alerts,_should_return_one_group_with_one_alert === RUN TestMergeV2AlertGroups/two_identical_groups_with_diverged_alerts,_should_return_one_group_with_the_newer_alert === RUN TestMergeV2AlertGroups/two_sets_of_identical_groups_with_single_alerts,_should_merge_all_into_two_groups_ === RUN TestMergeV2AlertGroups/many_unordered_groups,_should_return_groups_ordered_by_labels_then_receiver --- PASS: TestMergeV2AlertGroups (0.01s) --- PASS: TestMergeV2AlertGroups/no_groups,_should_return_no_groups (0.00s) --- PASS: TestMergeV2AlertGroups/one_group_with_one_alert,_should_return_one_group (0.00s) --- PASS: TestMergeV2AlertGroups/two_groups_with_different_labels,_should_return_two_groups (0.00s) --- PASS: TestMergeV2AlertGroups/two_groups_with_different_receiver,_should_return_two_groups (0.00s) --- PASS: TestMergeV2AlertGroups/two_identical_groups_with_different_alerts,_should_return_one_group_with_two_alerts (0.00s) --- PASS: TestMergeV2AlertGroups/two_identical_groups_with_identical_alerts,_should_return_one_group_with_one_alert (0.00s) --- PASS: TestMergeV2AlertGroups/two_identical_groups_with_diverged_alerts,_should_return_one_group_with_the_newer_alert (0.00s) --- PASS: TestMergeV2AlertGroups/two_sets_of_identical_groups_with_single_alerts,_should_merge_all_into_two_groups_ (0.00s) --- PASS: TestMergeV2AlertGroups/many_unordered_groups,_should_return_groups_ordered_by_labels_then_receiver (0.00s) === RUN TestV2Alerts --- PASS: TestV2Alerts (0.00s) === RUN TestMergeV2Alerts === RUN TestMergeV2Alerts/no_alerts,_should_return_an_empty_list === RUN TestMergeV2Alerts/one_alert,_should_return_the_alert === RUN TestMergeV2Alerts/two_alerts,_should_return_two_alerts === RUN TestMergeV2Alerts/three_alerts,_should_return_three_alerts === RUN TestMergeV2Alerts/three_alerts_out_of_order,_should_return_three_alerts_in_fingerprint_order === RUN TestMergeV2Alerts/two_identical_alerts,_should_return_one_alert === RUN TestMergeV2Alerts/two_identical_alerts_plus_another,_should_return_two_alerts === RUN TestMergeV2Alerts/two_duplicates_out_of_sync_alerts,_should_return_newer_alert === RUN TestMergeV2Alerts/two_duplicates_out_of_sync_alerts_(newer_first),_should_return_newer_alert === RUN TestMergeV2Alerts/two_duplicates_plus_others,_should_return_newer_alert_and_others --- PASS: TestMergeV2Alerts (0.00s) --- PASS: TestMergeV2Alerts/no_alerts,_should_return_an_empty_list (0.00s) --- PASS: TestMergeV2Alerts/one_alert,_should_return_the_alert (0.00s) --- PASS: TestMergeV2Alerts/two_alerts,_should_return_two_alerts (0.00s) --- PASS: TestMergeV2Alerts/three_alerts,_should_return_three_alerts (0.00s) --- PASS: TestMergeV2Alerts/three_alerts_out_of_order,_should_return_three_alerts_in_fingerprint_order (0.00s) --- PASS: TestMergeV2Alerts/two_identical_alerts,_should_return_one_alert (0.00s) --- PASS: TestMergeV2Alerts/two_identical_alerts_plus_another,_should_return_two_alerts (0.00s) --- PASS: TestMergeV2Alerts/two_duplicates_out_of_sync_alerts,_should_return_newer_alert (0.00s) --- PASS: TestMergeV2Alerts/two_duplicates_out_of_sync_alerts_(newer_first),_should_return_newer_alert (0.00s) --- PASS: TestMergeV2Alerts/two_duplicates_plus_others,_should_return_newer_alert_and_others (0.00s) === RUN TestV2SilenceId_ReturnsNewestSilence --- PASS: TestV2SilenceId_ReturnsNewestSilence (0.00s) === RUN TestV2SilenceID_InvalidDifferentIDs --- PASS: TestV2SilenceID_InvalidDifferentIDs (0.00s) === RUN TestV2Silences --- PASS: TestV2Silences (0.00s) === RUN TestMergeV2Silences === RUN TestMergeV2Silences/no_silences,_should_return_an_empty_list === RUN TestMergeV2Silences/one_silence,_should_return_the_silence === RUN TestMergeV2Silences/two_silences,_should_return_two_silences === RUN TestMergeV2Silences/three_silences,_should_return_three_silences === RUN TestMergeV2Silences/three_active_silences_out_of_order,_should_return_three_silences_in_expiry_order === RUN TestMergeV2Silences/two_identical_silences,_should_return_one_silence === RUN TestMergeV2Silences/two_identical_silences_plus_another,_should_return_two_silences === RUN TestMergeV2Silences/two_duplicates_out_of_sync_silences,_should_return_newer_silence === RUN TestMergeV2Silences/two_duplicates_out_of_sync_silences_(newer_first),_should_return_newer_silence === RUN TestMergeV2Silences/two_duplicates_plus_others,_should_return_newer_silence_and_others --- PASS: TestMergeV2Silences (0.00s) --- PASS: TestMergeV2Silences/no_silences,_should_return_an_empty_list (0.00s) --- PASS: TestMergeV2Silences/one_silence,_should_return_the_silence (0.00s) --- PASS: TestMergeV2Silences/two_silences,_should_return_two_silences (0.00s) --- PASS: TestMergeV2Silences/three_silences,_should_return_three_silences (0.00s) --- PASS: TestMergeV2Silences/three_active_silences_out_of_order,_should_return_three_silences_in_expiry_order (0.00s) --- PASS: TestMergeV2Silences/two_identical_silences,_should_return_one_silence (0.00s) --- PASS: TestMergeV2Silences/two_identical_silences_plus_another,_should_return_two_silences (0.00s) --- PASS: TestMergeV2Silences/two_duplicates_out_of_sync_silences,_should_return_newer_silence (0.00s) --- PASS: TestMergeV2Silences/two_duplicates_out_of_sync_silences_(newer_first),_should_return_newer_silence (0.00s) --- PASS: TestMergeV2Silences/two_duplicates_plus_others,_should_return_newer_silence_and_others (0.00s) PASS ok github.com/grafana/mimir/pkg/alertmanager/merger 0.337s === RUN TestNewApiWithoutSourceIPExtractor --- PASS: TestNewApiWithoutSourceIPExtractor (0.00s) === RUN TestNewApiWithSourceIPExtractor --- PASS: TestNewApiWithSourceIPExtractor (0.00s) === RUN TestNewApiWithInvalidSourceIPExtractor --- PASS: TestNewApiWithInvalidSourceIPExtractor (0.00s) === RUN TestApiGzip === RUN TestApiGzip/happy_case_gzip === RUN TestApiGzip/gzip_with_priority_header === RUN TestApiGzip/gzip_because_any_is_accepted === RUN TestApiGzip/no_gzip_because_no_header === RUN TestApiGzip/no_gzip_because_not_accepted === RUN TestApiGzip/no_gzip_because_small_payload === RUN TestApiGzip/forced_gzip_with_small_payload === RUN TestApiGzip/gzip_disabled_endpoint === RUN TestApiGzip/compressed_with_gzip --- PASS: TestApiGzip (0.08s) --- PASS: TestApiGzip/happy_case_gzip (0.02s) --- PASS: TestApiGzip/gzip_with_priority_header (0.01s) --- PASS: TestApiGzip/gzip_because_any_is_accepted (0.01s) --- PASS: TestApiGzip/no_gzip_because_no_header (0.01s) --- PASS: TestApiGzip/no_gzip_because_not_accepted (0.00s) --- PASS: TestApiGzip/no_gzip_because_small_payload (0.00s) --- PASS: TestApiGzip/forced_gzip_with_small_payload (0.01s) --- PASS: TestApiGzip/gzip_disabled_endpoint (0.00s) --- PASS: TestApiGzip/compressed_with_gzip (0.00s) === RUN TestIndexHandlerPrefix --- PASS: TestIndexHandlerPrefix (0.01s) === RUN TestIndexPageContent --- PASS: TestIndexPageContent (0.00s) === RUN TestConfigDiffHandler === RUN TestConfigDiffHandler/no_config_parameters_overridden === RUN TestConfigDiffHandler/slice_changed === RUN TestConfigDiffHandler/string_in_nested_struct_changed === RUN TestConfigDiffHandler/bool_in_nested_struct_changed === RUN TestConfigDiffHandler/test_invalid_input --- PASS: TestConfigDiffHandler (0.02s) --- PASS: TestConfigDiffHandler/no_config_parameters_overridden (0.00s) --- PASS: TestConfigDiffHandler/slice_changed (0.00s) --- PASS: TestConfigDiffHandler/string_in_nested_struct_changed (0.00s) --- PASS: TestConfigDiffHandler/bool_in_nested_struct_changed (0.00s) --- PASS: TestConfigDiffHandler/test_invalid_input (0.00s) === RUN TestConfigOverrideHandler --- PASS: TestConfigOverrideHandler (0.00s) PASS ok github.com/grafana/mimir/pkg/api 0.754s ? github.com/grafana/mimir/pkg/api/error [no test files] === RUN TestMultitenantCompactor_HandleBlockUpload_Create === RUN TestMultitenantCompactor_HandleBlockUpload_Create/missing_tenant_ID === RUN TestMultitenantCompactor_HandleBlockUpload_Create/missing_block_ID === RUN TestMultitenantCompactor_HandleBlockUpload_Create/invalid_block_ID === RUN TestMultitenantCompactor_HandleBlockUpload_Create/missing_body block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/malformed_body block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/invalid_file_path block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/missing_file_size block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/invalid_minTime block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/invalid_maxTime block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/maxTime_before_minTime block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/block_before_retention_period block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/invalid_version block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/ignore_retention_period_if_==_0 block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) block_upload_test.go:555: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/ignore_retention_period_if_<_0 block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) block_upload_test.go:555: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/invalid_compactor_shard_ID_label block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/failure_checking_for_complete_block block_upload_test.go:555: PASS: Exists(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/complete_block_already_exists block_upload_test.go:555: PASS: Exists(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/failure_uploading_meta_file block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) block_upload_test.go:555: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/block_upload_disabled === RUN TestMultitenantCompactor_HandleBlockUpload_Create/valid_request block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) block_upload_test.go:555: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_with_empty_compactor_shard_ID_label block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) block_upload_test.go:555: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_without_compactor_shard_ID_label block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) block_upload_test.go:555: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_with_different_block_ID_in_meta_file block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) block_upload_test.go:555: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_when_both_in-flight_meta_file_and_complete_meta_file_exist_in_object_storage === RUN TestMultitenantCompactor_HandleBlockUpload_Create/invalid_request_when_in-flight_meta_file_exists_in_object_storage === RUN TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_when_same_in-flight_meta_file_exists_in_object_storage === RUN TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_when_different_in-flight_meta_file_exists_in_object_storage --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create (0.15s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/missing_tenant_ID (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/missing_block_ID (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/invalid_block_ID (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/missing_body (0.01s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/malformed_body (0.01s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/invalid_file_path (0.01s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/missing_file_size (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/invalid_minTime (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/invalid_maxTime (0.01s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/maxTime_before_minTime (0.01s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/block_before_retention_period (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/invalid_version (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/ignore_retention_period_if_==_0 (0.01s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/ignore_retention_period_if_<_0 (0.01s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/invalid_compactor_shard_ID_label (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/failure_checking_for_complete_block (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/complete_block_already_exists (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/failure_uploading_meta_file (0.01s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/block_upload_disabled (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/valid_request (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_with_empty_compactor_shard_ID_label (0.01s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_without_compactor_shard_ID_label (0.02s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_with_different_block_ID_in_meta_file (0.01s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_when_both_in-flight_meta_file_and_complete_meta_file_exist_in_object_storage (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/invalid_request_when_in-flight_meta_file_exists_in_object_storage (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_when_same_in-flight_meta_file_exists_in_object_storage (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_when_different_in-flight_meta_file_exists_in_object_storage (0.00s) === RUN TestMultitenantCompactor_UploadBlockFile === RUN TestMultitenantCompactor_UploadBlockFile/without_tenant_ID === RUN TestMultitenantCompactor_UploadBlockFile/without_block_ID === RUN TestMultitenantCompactor_UploadBlockFile/invalid_block_ID === RUN TestMultitenantCompactor_UploadBlockFile/without_path === RUN TestMultitenantCompactor_UploadBlockFile/invalid_path === RUN TestMultitenantCompactor_UploadBlockFile/empty_file === RUN TestMultitenantCompactor_UploadBlockFile/attempt_block_metadata_file === RUN TestMultitenantCompactor_UploadBlockFile/attempt_in-flight_block_metadata_file === RUN TestMultitenantCompactor_UploadBlockFile/block_upload_disabled === RUN TestMultitenantCompactor_UploadBlockFile/complete_block_already_exists block_upload_test.go:967: PASS: Exists(string,string) === RUN TestMultitenantCompactor_UploadBlockFile/failure_checking_for_complete_block block_upload_test.go:967: PASS: Exists(string,string) === RUN TestMultitenantCompactor_UploadBlockFile/failure_checking_for_in-flight_meta_file block_upload_test.go:967: PASS: Exists(string,string) block_upload_test.go:967: PASS: Get(string,string) === RUN TestMultitenantCompactor_UploadBlockFile/missing_in-flight_meta_file block_upload_test.go:967: PASS: Exists(string,string) block_upload_test.go:967: PASS: Get(string,string) === RUN TestMultitenantCompactor_UploadBlockFile/file_upload_fails block_upload_test.go:967: PASS: Exists(string,string) block_upload_test.go:967: PASS: Get(string,string) block_upload_test.go:967: PASS: Get(string,string) block_upload_test.go:967: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_UploadBlockFile/invalid_file_size block_upload_test.go:967: PASS: Exists(string,string) block_upload_test.go:967: PASS: Get(string,string) block_upload_test.go:967: PASS: Get(string,string) === RUN TestMultitenantCompactor_UploadBlockFile/unexpected_file block_upload_test.go:967: PASS: Exists(string,string) block_upload_test.go:967: PASS: Get(string,string) block_upload_test.go:967: PASS: Get(string,string) === RUN TestMultitenantCompactor_UploadBlockFile/valid_request block_upload_test.go:967: PASS: Exists(string,string) block_upload_test.go:967: PASS: Get(string,string) block_upload_test.go:967: PASS: Get(string,string) block_upload_test.go:967: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_UploadBlockFile/multiple_sequential_uploads_of_same_file --- PASS: TestMultitenantCompactor_UploadBlockFile (0.09s) --- PASS: TestMultitenantCompactor_UploadBlockFile/without_tenant_ID (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/without_block_ID (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/invalid_block_ID (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/without_path (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/invalid_path (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/empty_file (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/attempt_block_metadata_file (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/attempt_in-flight_block_metadata_file (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/block_upload_disabled (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/complete_block_already_exists (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/failure_checking_for_complete_block (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/failure_checking_for_in-flight_meta_file (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/missing_in-flight_meta_file (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/file_upload_fails (0.02s) --- PASS: TestMultitenantCompactor_UploadBlockFile/invalid_file_size (0.01s) --- PASS: TestMultitenantCompactor_UploadBlockFile/unexpected_file (0.01s) --- PASS: TestMultitenantCompactor_UploadBlockFile/valid_request (0.01s) --- PASS: TestMultitenantCompactor_UploadBlockFile/multiple_sequential_uploads_of_same_file (0.00s) === RUN TestMultitenantCompactor_HandleBlockUpload_Complete === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/without_tenant_ID === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/without_block_ID === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/invalid_block_ID === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/block_upload_disabled === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/complete_block_already_exists block_upload_test.go:1264: PASS: Exists(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/checking_for_complete_block_fails block_upload_test.go:1264: PASS: Exists(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/missing_in-flight_meta_file block_upload_test.go:1264: PASS: Exists(string,string) block_upload_test.go:1264: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/downloading_in-flight_meta_file_fails block_upload_test.go:1264: PASS: Exists(string,string) block_upload_test.go:1264: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/corrupt_in-flight_meta_file block_upload_test.go:1264: PASS: Exists(string,string) block_upload_test.go:1264: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/uploading_meta_file_fails block_upload_test.go:1264: PASS: Exists(string,string) block_upload_test.go:1264: PASS: Get(string,string) block_upload_test.go:1264: PASS: Get(string,string) block_upload_test.go:1264: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/removing_in-flight_meta_file_fails block_upload_test.go:1264: PASS: Exists(string,string) block_upload_test.go:1264: PASS: Get(string,string) block_upload_test.go:1264: PASS: Get(string,string) block_upload_test.go:1264: PASS: Upload(string,string,string) block_upload_test.go:1264: PASS: Delete(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/valid_request block_upload_test.go:1264: PASS: Exists(string,string) block_upload_test.go:1264: PASS: Get(string,string) block_upload_test.go:1264: PASS: Get(string,string) block_upload_test.go:1264: PASS: Upload(string,string,string) block_upload_test.go:1264: PASS: Delete(string,string) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete (0.08s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/without_tenant_ID (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/without_block_ID (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/invalid_block_ID (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/block_upload_disabled (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/complete_block_already_exists (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/checking_for_complete_block_fails (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/missing_in-flight_meta_file (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/downloading_in-flight_meta_file_fails (0.01s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/corrupt_in-flight_meta_file (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/uploading_meta_file_fails (0.02s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/removing_in-flight_meta_file_fails (0.02s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/valid_request (0.02s) === RUN TestMultitenantCompactor_GetBlockUploadStateHandler === RUN TestMultitenantCompactor_GetBlockUploadStateHandler/stale_validation_file === RUN TestMultitenantCompactor_GetBlockUploadStateHandler/block_doesn't_exist === RUN TestMultitenantCompactor_GetBlockUploadStateHandler/complete_block === RUN TestMultitenantCompactor_GetBlockUploadStateHandler/upload_in_progress === RUN TestMultitenantCompactor_GetBlockUploadStateHandler/validating === RUN TestMultitenantCompactor_GetBlockUploadStateHandler/validation_failed --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler (0.00s) --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler/stale_validation_file (0.00s) --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler/block_doesn't_exist (0.00s) --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler/complete_block (0.00s) --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler/upload_in_progress (0.00s) --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler/validating (0.00s) --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler/validation_failed (0.00s) === RUN TestBlocksCleaner === RUN TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=0s === PAUSE TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=0s === RUN TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=2h0m0s === PAUSE TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=2h0m0s === RUN TestBlocksCleaner/concurrency=2,_tenant_deletion_delay=0s === PAUSE TestBlocksCleaner/concurrency=2,_tenant_deletion_delay=0s === RUN TestBlocksCleaner/concurrency=10,_tenant_deletion_delay=0s === PAUSE TestBlocksCleaner/concurrency=10,_tenant_deletion_delay=0s === CONT TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=0s === CONT TestBlocksCleaner/concurrency=2,_tenant_deletion_delay=0s === CONT TestBlocksCleaner/concurrency=10,_tenant_deletion_delay=0s === CONT TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=2h0m0s --- PASS: TestBlocksCleaner (0.00s) --- PASS: TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=2h0m0s (53.52s) --- PASS: TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=0s (54.53s) --- PASS: TestBlocksCleaner/concurrency=2,_tenant_deletion_delay=0s (55.20s) --- PASS: TestBlocksCleaner/concurrency=10,_tenant_deletion_delay=0s (56.68s) === RUN TestBlocksCleaner_ShouldContinueOnBlockDeletionFailure --- PASS: TestBlocksCleaner_ShouldContinueOnBlockDeletionFailure (13.92s) === RUN TestBlocksCleaner_ShouldRebuildBucketIndexOnCorruptedOne --- PASS: TestBlocksCleaner_ShouldRebuildBucketIndexOnCorruptedOne (17.65s) === RUN TestBlocksCleaner_ShouldRemoveMetricsForTenantsNotBelongingAnymoreToTheShard --- PASS: TestBlocksCleaner_ShouldRemoveMetricsForTenantsNotBelongingAnymoreToTheShard (24.20s) === RUN TestBlocksCleaner_ShouldNotCleanupUserThatDoesntBelongToShardAnymore --- PASS: TestBlocksCleaner_ShouldNotCleanupUserThatDoesntBelongToShardAnymore (9.36s) === RUN TestBlocksCleaner_ListBlocksOutsideRetentionPeriod --- PASS: TestBlocksCleaner_ListBlocksOutsideRetentionPeriod (16.68s) === RUN TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 10.252998ms logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 5.361999ms logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level debug component cleaner user user-1 msg applying retention retention 9h0m0s logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 5.331999ms logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 5.586999ms logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level debug component cleaner user user-1 msg applying retention retention 7h0m0s logger.go:22: level info component cleaner user user-1 msg applied retention: marking block for deletion block 01GTWJNH8RXB8HEHFYJCGN28DY maxTime 1678115033000 logger.go:22: level info component cleaner user user-1 msg block has been marked for deletion block 01GTWJNH8RXB8HEHFYJCGN28DY logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 8.090998ms logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 5.228999ms logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level debug component cleaner user user-1 msg applying retention retention 7h0m0s logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 5.330999ms logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 10.750998ms logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level debug component cleaner user user-1 msg applying retention retention 7h0m0s logger.go:22: level debug component cleaner user user-1 msg deleted file file 01GTWJNH8RXB8HEHFYJCGN28DY/meta.json bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod3505075263/001 logger.go:22: level debug component cleaner user user-1 msg deleted file file 01GTWJNH8RXB8HEHFYJCGN28DY/chunks/000001 bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod3505075263/001 logger.go:22: level debug component cleaner user user-1 msg deleted file file 01GTWJNH8RXB8HEHFYJCGN28DY/index bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod3505075263/001 logger.go:22: level debug component cleaner user user-1 msg deleted file file 01GTWJNH8RXB8HEHFYJCGN28DY/tombstones bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod3505075263/001 logger.go:22: level debug component cleaner user user-1 msg deleted file file 01GTWJNH8RXB8HEHFYJCGN28DY/deletion-mark.json bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod3505075263/001 logger.go:22: level info component cleaner user user-1 msg deleted block marked for deletion block 01GTWJNH8RXB8HEHFYJCGN28DY logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 9.199998ms logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 5.288999ms logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level debug component cleaner user user-1 msg applying retention retention 7h0m0s logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 4.981999ms logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level debug component cleaner user user-2 msg applying retention retention 5h0m0s logger.go:22: level info component cleaner user user-2 msg applied retention: marking block for deletion block 01GTWJNXH0NNHDG9F63VF40XW0 maxTime 1678115045000 logger.go:22: level info component cleaner user user-2 msg block has been marked for deletion block 01GTWJNXH0NNHDG9F63VF40XW0 logger.go:22: level info component cleaner user user-2 msg applied retention: marking block for deletion block 01GTWJP4DKK63FMBPPCC9HWQKQ maxTime 1678122252000 logger.go:22: level info component cleaner user user-2 msg block has been marked for deletion block 01GTWJP4DKK63FMBPPCC9HWQKQ logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GTWJNXH0NNHDG9F63VF40XW0/meta.json bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod3505075263/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GTWJNXH0NNHDG9F63VF40XW0/chunks/000001 bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod3505075263/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GTWJNXH0NNHDG9F63VF40XW0/index bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod3505075263/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GTWJNXH0NNHDG9F63VF40XW0/tombstones bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod3505075263/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GTWJNXH0NNHDG9F63VF40XW0/deletion-mark.json bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod3505075263/001 logger.go:22: level info component cleaner user user-2 msg deleted block marked for deletion block 01GTWJNXH0NNHDG9F63VF40XW0 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GTWJP4DKK63FMBPPCC9HWQKQ/meta.json bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod3505075263/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GTWJP4DKK63FMBPPCC9HWQKQ/chunks/000001 bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod3505075263/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GTWJP4DKK63FMBPPCC9HWQKQ/index bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod3505075263/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GTWJP4DKK63FMBPPCC9HWQKQ/tombstones bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod3505075263/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GTWJP4DKK63FMBPPCC9HWQKQ/deletion-mark.json bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod3505075263/001 logger.go:22: level info component cleaner user user-2 msg deleted block marked for deletion block 01GTWJP4DKK63FMBPPCC9HWQKQ logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 15.251997ms --- PASS: TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod (25.52s) === RUN TestBlocksCleaner_ShouldRemovePartialBlocksOutsideDelayPeriod logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level warn component cleaner user user-1 msg skipped partial block when updating bucket index block 01GTWJPA4K88JW58RPD16C3FGS logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 6.392998ms logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level warn component cleaner user user-1 msg skipped partial block when updating bucket index block 01GTWJPA4K88JW58RPD16C3FGS logger.go:22: level info component cleaner user user-1 msg stale partial block found: marking block for deletion block 01GTWJPA4K88JW58RPD16C3FGS last modified 2023-03-07 07:04:23.631319977 +0800 CST logger.go:22: level info component cleaner user user-1 msg block has been marked for deletion block 01GTWJPA4K88JW58RPD16C3FGS logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 8.630998ms --- PASS: TestBlocksCleaner_ShouldRemovePartialBlocksOutsideDelayPeriod (9.37s) === RUN TestBlocksCleaner_ShouldNotRemovePartialBlocksInsideDelayPeriod logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level warn component cleaner user user-1 msg skipped partial block when updating bucket index block 01GTWJPKBXS44ZSQ7YAHQ6XMR9 logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 6.182998ms logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level error component cleaner user user-2 msg skipped block with corrupted meta.json when updating bucket index block 01GTWJPQQG5CCP0DRCX88GTYVS err unmarshal block meta file 01GTWJPQQG5CCP0DRCX88GTYVS/meta.json: invalid character 'c' looking for beginning of value: meta.json corrupted logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 4.736999ms --- PASS: TestBlocksCleaner_ShouldNotRemovePartialBlocksInsideDelayPeriod (10.17s) === RUN TestBlocksCleaner_ShouldNotRemovePartialBlocksIfConfiguredDelayIsInvalid --- PASS: TestBlocksCleaner_ShouldNotRemovePartialBlocksIfConfiguredDelayIsInvalid (5.58s) === RUN TestFindMostRecentModifiedTimeForBlock --- PASS: TestFindMostRecentModifiedTimeForBlock (5.74s) === RUN TestSyncer_GarbageCollect_e2e === PAUSE TestSyncer_GarbageCollect_e2e === RUN TestGroupCompactE2E === PAUSE TestGroupCompactE2E === RUN TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly === PAUSE TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly === RUN TestGroupKey === RUN TestGroupKey/#00 === RUN TestGroupKey/#01 === RUN TestGroupKey/#02 === RUN TestGroupKey/#03 --- PASS: TestGroupKey (0.00s) --- PASS: TestGroupKey/#00 (0.00s) --- PASS: TestGroupKey/#01 (0.00s) --- PASS: TestGroupKey/#02 (0.00s) --- PASS: TestGroupKey/#03 (0.00s) === RUN TestGroupMaxMinTime --- PASS: TestGroupMaxMinTime (0.00s) === RUN TestFilterOwnJobs === RUN TestFilterOwnJobs/should_return_all_planned_jobs_if_the_compactor_instance_owns_all_of_them === RUN TestFilterOwnJobs/should_return_no_jobs_if_the_compactor_instance_owns_none_of_them === RUN TestFilterOwnJobs/should_return_some_jobs_if_the_compactor_instance_owns_some_of_them --- PASS: TestFilterOwnJobs (0.01s) --- PASS: TestFilterOwnJobs/should_return_all_planned_jobs_if_the_compactor_instance_owns_all_of_them (0.00s) --- PASS: TestFilterOwnJobs/should_return_no_jobs_if_the_compactor_instance_owns_none_of_them (0.00s) --- PASS: TestFilterOwnJobs/should_return_some_jobs_if_the_compactor_instance_owns_some_of_them (0.00s) === RUN TestBlockMaxTimeDeltas --- PASS: TestBlockMaxTimeDeltas (0.00s) === RUN TestNoCompactionMarkFilter === RUN TestNoCompactionMarkFilter/filter_with_no_deletion_of_blocks_marked_for_no-compaction === RUN TestNoCompactionMarkFilter/filter_with_deletion_enabled === RUN TestNoCompactionMarkFilter/filter_with_deletion_enabled,_but_canceled_context === RUN TestNoCompactionMarkFilter/filtering_block_with_wrong_marker_version --- PASS: TestNoCompactionMarkFilter (0.01s) --- PASS: TestNoCompactionMarkFilter/filter_with_no_deletion_of_blocks_marked_for_no-compaction (0.01s) --- PASS: TestNoCompactionMarkFilter/filter_with_deletion_enabled (0.00s) --- PASS: TestNoCompactionMarkFilter/filter_with_deletion_enabled,_but_canceled_context (0.00s) --- PASS: TestNoCompactionMarkFilter/filtering_block_with_wrong_marker_version (0.00s) === RUN TestConvertCompactionResultToForEachJobs --- PASS: TestConvertCompactionResultToForEachJobs (0.00s) === RUN TestConfig_ShouldSupportYamlConfig --- PASS: TestConfig_ShouldSupportYamlConfig (0.00s) === RUN TestConfig_ShouldSupportCliFlags --- PASS: TestConfig_ShouldSupportCliFlags (0.00s) === RUN TestConfig_Validate === RUN TestConfig_Validate/should_fail_on_invalid_value_of_max-closing-blocks-concurrency === RUN TestConfig_Validate/should_fail_on_invalid_value_of_symbols-flushers-concurrency === RUN TestConfig_Validate/should_pass_with_the_default_config === RUN TestConfig_Validate/should_pass_with_only_1_block_range_period === RUN TestConfig_Validate/should_fail_with_non_divisible_block_range_periods === RUN TestConfig_Validate/should_fail_on_unknown_compaction_jobs_order === RUN TestConfig_Validate/should_fail_on_invalid_value_of_max-opening-blocks-concurrency --- PASS: TestConfig_Validate (0.03s) --- PASS: TestConfig_Validate/should_fail_on_invalid_value_of_max-closing-blocks-concurrency (0.00s) --- PASS: TestConfig_Validate/should_fail_on_invalid_value_of_symbols-flushers-concurrency (0.02s) --- PASS: TestConfig_Validate/should_pass_with_the_default_config (0.00s) --- PASS: TestConfig_Validate/should_pass_with_only_1_block_range_period (0.00s) --- PASS: TestConfig_Validate/should_fail_with_non_divisible_block_range_periods (0.00s) --- PASS: TestConfig_Validate/should_fail_on_unknown_compaction_jobs_order (0.00s) --- PASS: TestConfig_Validate/should_fail_on_invalid_value_of_max-opening-blocks-concurrency (0.00s) === RUN TestMultitenantCompactor_ShouldDoNothingOnNoUserBlocks === PAUSE TestMultitenantCompactor_ShouldDoNothingOnNoUserBlocks === RUN TestMultitenantCompactor_ShouldRetryCompactionOnFailureWhileDiscoveringUsersFromBucket === PAUSE TestMultitenantCompactor_ShouldRetryCompactionOnFailureWhileDiscoveringUsersFromBucket === RUN TestMultitenantCompactor_ShouldIncrementCompactionErrorIfFailedToCompactASingleTenant === PAUSE TestMultitenantCompactor_ShouldIncrementCompactionErrorIfFailedToCompactASingleTenant === RUN TestMultitenantCompactor_ShouldIterateOverUsersAndRunCompaction === PAUSE TestMultitenantCompactor_ShouldIterateOverUsersAndRunCompaction === RUN TestMultitenantCompactor_ShouldStopCompactingTenantOnReachingMaxCompactionTime === PAUSE TestMultitenantCompactor_ShouldStopCompactingTenantOnReachingMaxCompactionTime === RUN TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForDeletion === PAUSE TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForDeletion === RUN TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForNoCompaction === PAUSE TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForNoCompaction === RUN TestMultitenantCompactor_ShouldNotCompactBlocksForUsersMarkedForDeletion === PAUSE TestMultitenantCompactor_ShouldNotCompactBlocksForUsersMarkedForDeletion === RUN TestMultitenantCompactor_ShouldCompactAllUsersOnShardingEnabledButOnlyOneInstanceRunning === PAUSE TestMultitenantCompactor_ShouldCompactAllUsersOnShardingEnabledButOnlyOneInstanceRunning === RUN TestMultitenantCompactor_ShouldCompactOnlyUsersOwnedByTheInstanceOnShardingEnabledAndMultipleInstancesRunning === PAUSE TestMultitenantCompactor_ShouldCompactOnlyUsersOwnedByTheInstanceOnShardingEnabledAndMultipleInstancesRunning === RUN TestMultitenantCompactor_ShouldSkipCompactionForJobsNoMoreOwnedAfterPlanning === PAUSE TestMultitenantCompactor_ShouldSkipCompactionForJobsNoMoreOwnedAfterPlanning === RUN TestMultitenantCompactor_DeleteLocalSyncFiles --- PASS: TestMultitenantCompactor_DeleteLocalSyncFiles (6.48s) === RUN TestMultitenantCompactor_ShouldFailCompactionOnTimeout === PAUSE TestMultitenantCompactor_ShouldFailCompactionOnTimeout === RUN TestOwnUser === RUN TestOwnUser/5_compactors,_sharding_enabled,_no_compactor_shard_size === PAUSE TestOwnUser/5_compactors,_sharding_enabled,_no_compactor_shard_size === RUN TestOwnUser/10_compactors,_sharding_enabled,_with_non-zero_shard_sizes === PAUSE TestOwnUser/10_compactors,_sharding_enabled,_with_non-zero_shard_sizes === RUN TestOwnUser/10_compactors,_sharding_enabled,_with_zero_shard_size === PAUSE TestOwnUser/10_compactors,_sharding_enabled,_with_zero_shard_size === CONT TestOwnUser/5_compactors,_sharding_enabled,_no_compactor_shard_size === CONT TestOwnUser/10_compactors,_sharding_enabled,_with_zero_shard_size === CONT TestOwnUser/10_compactors,_sharding_enabled,_with_non-zero_shard_sizes --- PASS: TestOwnUser (0.00s) --- PASS: TestOwnUser/10_compactors,_sharding_enabled,_with_zero_shard_size (5.37s) --- PASS: TestOwnUser/5_compactors,_sharding_enabled,_no_compactor_shard_size (5.42s) --- PASS: TestOwnUser/10_compactors,_sharding_enabled,_with_non-zero_shard_sizes (6.16s) === RUN TestMultitenantCompactor_OutOfOrderCompaction --- PASS: TestMultitenantCompactor_OutOfOrderCompaction (12.98s) === RUN TestSortJobsBySmallestRangeOldestBlocksFirst === RUN TestSortJobsBySmallestRangeOldestBlocksFirst/should_do_nothing_on_empty_input === RUN TestSortJobsBySmallestRangeOldestBlocksFirst/should_sort_jobs_by_smallest_range,_oldest_blocks_first === RUN TestSortJobsBySmallestRangeOldestBlocksFirst/split_jobs_are_always_sorted_first --- PASS: TestSortJobsBySmallestRangeOldestBlocksFirst (0.00s) --- PASS: TestSortJobsBySmallestRangeOldestBlocksFirst/should_do_nothing_on_empty_input (0.00s) --- PASS: TestSortJobsBySmallestRangeOldestBlocksFirst/should_sort_jobs_by_smallest_range,_oldest_blocks_first (0.00s) --- PASS: TestSortJobsBySmallestRangeOldestBlocksFirst/split_jobs_are_always_sorted_first (0.00s) === RUN TestSortJobsByNewestBlocksFirst === RUN TestSortJobsByNewestBlocksFirst/should_do_nothing_on_empty_input job_sorting_test.go:127: sorted jobs: === RUN TestSortJobsByNewestBlocksFirst/should_sort_jobs_by_newest_blocks_first job_sorting_test.go:127: sorted jobs: job_sorting_test.go:129: - (minTime: 40 maxTime: 80) job_sorting_test.go:129: - (minTime: 10 maxTime: 30) job_sorting_test.go:129: - (minTime: 10 maxTime: 20) === RUN TestSortJobsByNewestBlocksFirst/should_give_precedence_to_smaller_time_ranges_in_case_of_multiple_jobs_with_the_same_max_time job_sorting_test.go:127: sorted jobs: job_sorting_test.go:129: - (minTime: 30 maxTime: 40) job_sorting_test.go:129: - (minTime: 10 maxTime: 40) === RUN TestSortJobsByNewestBlocksFirst/should_give_precedence_to_newest_blocks_over_smaller_time_ranges job_sorting_test.go:127: sorted jobs: job_sorting_test.go:129: - (minTime: 10 maxTime: 40) job_sorting_test.go:129: - (minTime: 10 maxTime: 30) job_sorting_test.go:129: - (minTime: 10 maxTime: 20) --- PASS: TestSortJobsByNewestBlocksFirst (0.02s) --- PASS: TestSortJobsByNewestBlocksFirst/should_do_nothing_on_empty_input (0.01s) --- PASS: TestSortJobsByNewestBlocksFirst/should_sort_jobs_by_newest_blocks_first (0.00s) --- PASS: TestSortJobsByNewestBlocksFirst/should_give_precedence_to_smaller_time_ranges_in_case_of_multiple_jobs_with_the_same_max_time (0.00s) --- PASS: TestSortJobsByNewestBlocksFirst/should_give_precedence_to_newest_blocks_over_smaller_time_ranges (0.00s) === RUN TestLabelRemoverFilter === RUN TestLabelRemoverFilter/should_remove_configured_labels === RUN TestLabelRemoverFilter/should_remove_configured_labels_2 --- PASS: TestLabelRemoverFilter (0.00s) --- PASS: TestLabelRemoverFilter/should_remove_configured_labels (0.00s) --- PASS: TestLabelRemoverFilter/should_remove_configured_labels_2 (0.00s) === RUN TestShardAwareDeduplicateFilter_Filter === RUN TestShardAwareDeduplicateFilter_Filter/two_compacted_blocks_with_overlapping_sources === RUN TestShardAwareDeduplicateFilter_Filter/4_non_compacted_blocks_and_compacted_block_of_level_2_in_bucket === RUN TestShardAwareDeduplicateFilter_Filter/two_blocks_merged_and_split,_with_two_shards === RUN TestShardAwareDeduplicateFilter_Filter/blocks_with_same_sources_and_different_resolutions === RUN TestShardAwareDeduplicateFilter_Filter/compacted_blocks_of_level_3_with_overlapping_sources_of_different_length_and_different_resolutions === RUN TestShardAwareDeduplicateFilter_Filter/four_base_blocks_merged_and_split_into_2_separate_shards,_plus_another_level,_with_various_resolutions === RUN TestShardAwareDeduplicateFilter_Filter/two_blocks_merged_and_split_into_two,_one_shard_missing === RUN TestShardAwareDeduplicateFilter_Filter/four_base_blocks_merged_and_split_into_2_separate_shards === RUN TestShardAwareDeduplicateFilter_Filter/two_compacted_blocks_with_same_sources === RUN TestShardAwareDeduplicateFilter_Filter/compacted_blocks_with_overlapping_sources === RUN TestShardAwareDeduplicateFilter_Filter/compacted_blocks_of_level_3_with_overlapping_sources_of_different_length === RUN TestShardAwareDeduplicateFilter_Filter/compacted_blocks_with_overlapping_sources_and_different_resolutions === RUN TestShardAwareDeduplicateFilter_Filter/block_with_invalid_shardID_cannot_'include'_its_source_blocks === RUN TestShardAwareDeduplicateFilter_Filter/when_invalid_shard_IDs_present,_no_deduplication_happens_for_source_blocks === RUN TestShardAwareDeduplicateFilter_Filter/3_non_compacted_blocks_in_bucket === RUN TestShardAwareDeduplicateFilter_Filter/compacted_block_without_sources_in_bucket === RUN TestShardAwareDeduplicateFilter_Filter/3_compacted_blocks_of_level_2_and_one_compacted_block_of_level_3_in_bucket === RUN TestShardAwareDeduplicateFilter_Filter/compacted_blocks_of_level_3_with_overlapping_sources_of_equal_length === RUN TestShardAwareDeduplicateFilter_Filter/two_blocks_merged_and_split,_with_single_shard === RUN TestShardAwareDeduplicateFilter_Filter/four_base_blocks_merged_and_split_into_2_separate_shards,_plus_another_level --- PASS: TestShardAwareDeduplicateFilter_Filter (0.03s) --- PASS: TestShardAwareDeduplicateFilter_Filter/two_compacted_blocks_with_overlapping_sources (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/4_non_compacted_blocks_and_compacted_block_of_level_2_in_bucket (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/two_blocks_merged_and_split,_with_two_shards (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/blocks_with_same_sources_and_different_resolutions (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/compacted_blocks_of_level_3_with_overlapping_sources_of_different_length_and_different_resolutions (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/four_base_blocks_merged_and_split_into_2_separate_shards,_plus_another_level,_with_various_resolutions (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/two_blocks_merged_and_split_into_two,_one_shard_missing (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/four_base_blocks_merged_and_split_into_2_separate_shards (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/two_compacted_blocks_with_same_sources (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/compacted_blocks_with_overlapping_sources (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/compacted_blocks_of_level_3_with_overlapping_sources_of_different_length (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/compacted_blocks_with_overlapping_sources_and_different_resolutions (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/block_with_invalid_shardID_cannot_'include'_its_source_blocks (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/when_invalid_shard_IDs_present,_no_deduplication_happens_for_source_blocks (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/3_non_compacted_blocks_in_bucket (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/compacted_block_without_sources_in_bucket (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/3_compacted_blocks_of_level_2_and_one_compacted_block_of_level_3_in_bucket (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/compacted_blocks_of_level_3_with_overlapping_sources_of_equal_length (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/two_blocks_merged_and_split,_with_single_shard (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/four_base_blocks_merged_and_split_into_2_separate_shards,_plus_another_level (0.00s) === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=root3 ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=cleaner run_id=1678143926 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=5.340999ms level=info component=cleaner run_id=1678143926 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=6.888998ms duration_ms=6 cached=2 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job="stage: split, range start: 7200000, range end: 14400000, shard: 1_of_1, blocks: 01GTWJR1FB59BP55YAB4KGKJ36 (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC),01GTWJR718VZACV8VE35QTFDD5 (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GTWJR718VZACV8VE35QTFDD5 (min time: 7200000, max time: 14400000) 01GTWJR1FB59BP55YAB4KGKJ36 (min time: 7200000, max time: 14400000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GTWJR1FB59BP55YAB4KGKJ36/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GTWJR718VZACV8VE35QTFDD5/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3367260562/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GTWJR718VZACV8VE35QTFDD5 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3367260562/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GTWJR1FB59BP55YAB4KGKJ36]" duration=11.848997ms duration_ms=11 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01GTWJRCJCXW570262BA7NHVC7 sources="[01GTWJR718VZACV8VE35QTFDD5 01GTWJR1FB59BP55YAB4KGKJ36]" duration=7.087724421s shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01GTWJRCJCQQM9A7AXK0CXSY64 sources="[01GTWJR718VZACV8VE35QTFDD5 01GTWJR1FB59BP55YAB4KGKJ36]" duration=7.088122421s shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new="[01GTWJRCJCXW570262BA7NHVC7 01GTWJRCJCQQM9A7AXK0CXSY64]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3367260562/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GTWJR718VZACV8VE35QTFDD5 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3367260562/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GTWJR1FB59BP55YAB4KGKJ36]" duration=7.08867742s duration_ms=7088 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3367260562/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GTWJRCJCQQM9A7AXK0CXSY64/chunks/000001 dst=01GTWJRCJCQQM9A7AXK0CXSY64/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3367260562/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3367260562/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GTWJRCJCQQM9A7AXK0CXSY64/index dst=01GTWJRCJCQQM9A7AXK0CXSY64/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3367260562/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GTWJRCJCQQM9A7AXK0CXSY64 duration=3.317999ms duration_ms=3 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3367260562/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GTWJRCJCXW570262BA7NHVC7/chunks/000001 dst=01GTWJRCJCXW570262BA7NHVC7/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3367260562/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3367260562/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GTWJRCJCXW570262BA7NHVC7/index dst=01GTWJRCJCXW570262BA7NHVC7/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3367260562/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GTWJRCJCXW570262BA7NHVC7 duration=4.009999ms duration_ms=4 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=7.617999ms duration_ms=7 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GTWJR718VZACV8VE35QTFDD5 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GTWJR718VZACV8VE35QTFDD5 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GTWJR1FB59BP55YAB4KGKJ36 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GTWJR1FB59BP55YAB4KGKJ36 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=7.113927415s duration_ms=7113 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=4.161999ms duration_ms=4 cached=4 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=8.567998ms duration_ms=8 cached=4 returned=2 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=root3 ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1678143942 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=6.309999ms duration_ms=6 cached=2 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job="stage: split, range start: 7200000, range end: 14400000, shard: 1_of_1, blocks: 01GTWJRKVJRPY4E6MYT9VJ25SE (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC),01GTWJRQYN4KZ4TJX3EQB81DX0 (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GTWJRKVJRPY4E6MYT9VJ25SE (min time: 7200000, max time: 14400000) 01GTWJRQYN4KZ4TJX3EQB81DX0 (min time: 7200000, max time: 14400000)]" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=12.656997ms level=info component=cleaner run_id=1678143942 msg="successfully completed blocks cleanup and maintenance" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GTWJRKVJRPY4E6MYT9VJ25SE/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GTWJRQYN4KZ4TJX3EQB81DX0/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split2192328851/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GTWJRKVJRPY4E6MYT9VJ25SE /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split2192328851/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GTWJRQYN4KZ4TJX3EQB81DX0]" duration=14.412997ms duration_ms=14 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01GTWJRW306TQHN8X8S9Q0YJTM sources="[01GTWJRKVJRPY4E6MYT9VJ25SE 01GTWJRQYN4KZ4TJX3EQB81DX0]" duration=6.374699579s shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01GTWJRW30DB16JP11ZYMDQ2AD sources="[01GTWJRKVJRPY4E6MYT9VJ25SE 01GTWJRQYN4KZ4TJX3EQB81DX0]" duration=6.374881579s shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new="[01GTWJRW306TQHN8X8S9Q0YJTM 01GTWJRW30DB16JP11ZYMDQ2AD]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split2192328851/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GTWJRKVJRPY4E6MYT9VJ25SE /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split2192328851/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GTWJRQYN4KZ4TJX3EQB81DX0]" duration=6.375187579s duration_ms=6375 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split2192328851/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GTWJRW30DB16JP11ZYMDQ2AD/chunks/000001 dst=01GTWJRW30DB16JP11ZYMDQ2AD/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split2192328851/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split2192328851/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GTWJRW30DB16JP11ZYMDQ2AD/index dst=01GTWJRW30DB16JP11ZYMDQ2AD/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split2192328851/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GTWJRW30DB16JP11ZYMDQ2AD duration=3.096999ms duration_ms=3 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split2192328851/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GTWJRW306TQHN8X8S9Q0YJTM/chunks/000001 dst=01GTWJRW306TQHN8X8S9Q0YJTM/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split2192328851/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split2192328851/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GTWJRW306TQHN8X8S9Q0YJTM/index dst=01GTWJRW306TQHN8X8S9Q0YJTM/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split2192328851/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GTWJRW306TQHN8X8S9Q0YJTM duration=3.986999ms duration_ms=3 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=12.262997ms duration_ms=12 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GTWJRKVJRPY4E6MYT9VJ25SE level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GTWJRKVJRPY4E6MYT9VJ25SE level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GTWJRQYN4KZ4TJX3EQB81DX0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GTWJRQYN4KZ4TJX3EQB81DX0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=6.406840573s duration_ms=6406 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=8.249998ms duration_ms=8 cached=4 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=12.127997ms duration_ms=12 cached=4 returned=2 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=root3 ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1678143961 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=11.728997ms duration_ms=11 cached=3 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job="stage: split, range start: 0, range end: 7200000, shard: 1_of_1, blocks: 01GTWJS2FAK0RCV2GC9Q66XJME (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01GTWJS6E6HVEAW2FAW70WVGH8 (min time: 1970-01-01 00:01:00 +0000 UTC, max time: 1970-01-01 00:07:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=17.104997ms level=info component=cleaner run_id=1678143961 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GTWJS2FAK0RCV2GC9Q66XJME (min time: 0, max time: 300000) 01GTWJS6E6HVEAW2FAW70WVGH8 (min time: 60000, max time: 420000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GTWJS2FAK0RCV2GC9Q66XJME/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GTWJS6E6HVEAW2FAW70WVGH8/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split2683842769/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJS2FAK0RCV2GC9Q66XJME /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split2683842769/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJS6E6HVEAW2FAW70WVGH8]" duration=16.082996ms duration_ms=16 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=420000 ulid=01GTWJSE5YS8EN61FQV00BH3AY sources="[01GTWJS2FAK0RCV2GC9Q66XJME 01GTWJS6E6HVEAW2FAW70WVGH8]" duration=8.081072199s shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=420000 ulid=01GTWJSE5YQV04KJWDXDXBBC1D sources="[01GTWJS2FAK0RCV2GC9Q66XJME 01GTWJS6E6HVEAW2FAW70WVGH8]" duration=8.081274199s shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="compacted blocks" new="[01GTWJSE5YS8EN61FQV00BH3AY 01GTWJSE5YQV04KJWDXDXBBC1D]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split2683842769/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJS2FAK0RCV2GC9Q66XJME /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split2683842769/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJS6E6HVEAW2FAW70WVGH8]" duration=8.081659199s duration_ms=8081 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split2683842769/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJSE5YQV04KJWDXDXBBC1D/chunks/000001 dst=01GTWJSE5YQV04KJWDXDXBBC1D/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split2683842769/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split2683842769/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJSE5YQV04KJWDXDXBBC1D/index dst=01GTWJSE5YQV04KJWDXDXBBC1D/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split2683842769/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split2683842769/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJSE5YS8EN61FQV00BH3AY/chunks/000001 dst=01GTWJSE5YS8EN61FQV00BH3AY/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split2683842769/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split2683842769/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJSE5YS8EN61FQV00BH3AY/index dst=01GTWJSE5YS8EN61FQV00BH3AY/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split2683842769/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded block" result_block=01GTWJSE5YQV04KJWDXDXBBC1D duration=6.456999ms duration_ms=6 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded block" result_block=01GTWJSE5YS8EN61FQV00BH3AY duration=6.741999ms duration_ms=6 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=9.347998ms duration_ms=9 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GTWJS2FAK0RCV2GC9Q66XJME level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="block has been marked for deletion" block=01GTWJS2FAK0RCV2GC9Q66XJME level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GTWJS6E6HVEAW2FAW70WVGH8 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="block has been marked for deletion" block=01GTWJS6E6HVEAW2FAW70WVGH8 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="compaction job succeeded" duration=8.119033191s duration_ms=8119 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=4.320999ms duration_ms=4 cached=5 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=18.054996ms duration_ms=18 cached=5 returned=3 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/smaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=root3 ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1678144029 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=12.086997ms duration_ms=12 cached=10 returned=10 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job="stage: merge, range start: 14400000, range end: 28800000, shard: 1_of_2, blocks: 01GTWJTBX5KKQ6H4P8VQED3HTG (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 06:00:00 +0000 UTC),01GTWJTQ52YCFQ4YP571T1Q050 (min time: 1970-01-01 06:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job="stage: merge, range start: 14400000, range end: 28800000, shard: 2_of_2, blocks: 01GTWJTHNQ8G6HFXA52Q7TMYS8 (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 06:00:00 +0000 UTC),01GTWJTXHJM3NE9NZFJ6PNX5X8 (min time: 1970-01-01 06:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 job="stage: split, range start: 28800000, range end: 36000000, shard: 1_of_1, blocks: 01GTWJV4B6A5Q6MEPCJAGTR8CB (min time: 1970-01-01 08:00:00 +0000 UTC, max time: 1970-01-01 10:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 job="stage: split, range start: 50400000, range end: 57600000, shard: 1_of_1, blocks: 01GTWJVB514Q7KT4RWKY9VJY4H (min time: 1970-01-01 14:00:00 +0000 UTC, max time: 1970-01-01 16:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job="stage: split, range start: 7200000, range end: 14400000, shard: 1_of_1, blocks: 01GTWJT0X8N996XQR5X4BXWCA9 (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC),01GTWJT6E1KN11CAGGH6KNFGFN (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GTWJT0X8N996XQR5X4BXWCA9 (min time: 7200000, max time: 14400000) 01GTWJT6E1KN11CAGGH6KNFGFN (min time: 7200000, max time: 14400000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GTWJT0X8N996XQR5X4BXWCA9/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GTWJT6E1KN11CAGGH6KNFGFN/meta.json level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=28.347994ms level=info component=cleaner run_id=1678144029 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges651038108/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GTWJT0X8N996XQR5X4BXWCA9 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges651038108/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GTWJT6E1KN11CAGGH6KNFGFN]" duration=13.340997ms duration_ms=13 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01GTWJVH4AGEGMZGM6FBYPRW67 sources="[01GTWJT0X8N996XQR5X4BXWCA9 01GTWJT6E1KN11CAGGH6KNFGFN]" duration=10.687996618s shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01GTWJVH4AARTYX9PJ564M0VKG sources="[01GTWJT0X8N996XQR5X4BXWCA9 01GTWJT6E1KN11CAGGH6KNFGFN]" duration=10.688461618s shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new="[01GTWJVH4AGEGMZGM6FBYPRW67 01GTWJVH4AARTYX9PJ564M0VKG]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges651038108/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GTWJT0X8N996XQR5X4BXWCA9 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges651038108/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GTWJT6E1KN11CAGGH6KNFGFN]" duration=10.689104618s duration_ms=10689 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges651038108/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GTWJVH4AARTYX9PJ564M0VKG/chunks/000001 dst=01GTWJVH4AARTYX9PJ564M0VKG/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges651038108/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges651038108/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GTWJVH4AARTYX9PJ564M0VKG/index dst=01GTWJVH4AARTYX9PJ564M0VKG/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges651038108/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GTWJVH4AARTYX9PJ564M0VKG duration=3.920999ms duration_ms=3 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges651038108/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GTWJVH4AGEGMZGM6FBYPRW67/chunks/000001 dst=01GTWJVH4AGEGMZGM6FBYPRW67/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges651038108/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges651038108/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GTWJVH4AGEGMZGM6FBYPRW67/index dst=01GTWJVH4AGEGMZGM6FBYPRW67/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges651038108/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GTWJVH4AGEGMZGM6FBYPRW67 duration=4.820999ms duration_ms=4 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=12.437997ms duration_ms=12 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GTWJT0X8N996XQR5X4BXWCA9 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GTWJT0X8N996XQR5X4BXWCA9 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GTWJT6E1KN11CAGGH6KNFGFN level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GTWJT6E1KN11CAGGH6KNFGFN level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=10.720863611s duration_ms=10720 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=1 plan="[01GTWJV4B6A5Q6MEPCJAGTR8CB (min time: 28800000, max time: 36000000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GTWJV4B6A5Q6MEPCJAGTR8CB/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=1 plan=[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges651038108/001/compact/0@17241709254077376921-split-1_of_1-28800000-36000000/01GTWJV4B6A5Q6MEPCJAGTR8CB] duration=18.109996ms duration_ms=18 split_merge_compactor_test.go:640: expected , got Diff: --- metric output does not match expectation; want +++ got: @@ -2,3 +2,3 @@ # TYPE cortex_compactor_runs_completed_total counter -cortex_compactor_runs_completed_total 1 +cortex_compactor_runs_completed_total 0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor level=error component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="compaction job failed" duration=4.607280973s duration_ms=4607 err="compact blocks [/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges651038108/001/compact/0@17241709254077376921-split-1_of_1-28800000-36000000/01GTWJV4B6A5Q6MEPCJAGTR8CB]: 3 errors: populate block: context canceled; context canceled; context canceled" level=error component=compactor msg="failed to compact user blocks" user=user-1 err="compaction: group 0@17241709254077376921-split-1_of_1-28800000-36000000: compact blocks [/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges651038108/001/compact/0@17241709254077376921-split-1_of_1-28800000-36000000/01GTWJV4B6A5Q6MEPCJAGTR8CB]: 3 errors: populate block: context canceled; context canceled; context canceled" === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together level=info component=compactor msg="instance not found in the ring" instance=root3 ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=cleaner run_id=1678144064 msg="started blocks cleanup and maintenance" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=5.365999ms duration_ms=5 cached=3 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job="stage: split, range start: 0, range end: 7200000, shard: 1_of_1, blocks: 01GTWJW0819ZTCRZ2DD7YKKBG3 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01GTWJW5EWHRYWGJ09A309FMT1 (min time: 1970-01-01 00:01:00 +0000 UTC, max time: 1970-01-01 00:07:00 +0000 UTC),01GTWJWC6NREBNJ5YWKA73NWEF (min time: 1970-01-01 01:00:00 +0000 UTC, max time: 1970-01-01 02:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=3 plan="[01GTWJW0819ZTCRZ2DD7YKKBG3 (min time: 0, max time: 300000) 01GTWJW5EWHRYWGJ09A309FMT1 (min time: 60000, max time: 420000) 01GTWJWC6NREBNJ5YWKA73NWEF (min time: 3600000, max time: 7200000)]" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=8.888998ms level=info component=cleaner run_id=1678144064 msg="successfully completed blocks cleanup and maintenance" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GTWJWC6NREBNJ5YWKA73NWEF/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GTWJW5EWHRYWGJ09A309FMT1/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GTWJW0819ZTCRZ2DD7YKKBG3/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=3 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3469273953/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJW0819ZTCRZ2DD7YKKBG3 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3469273953/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJW5EWHRYWGJ09A309FMT1 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3469273953/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJWC6NREBNJ5YWKA73NWEF]" duration=13.114997ms duration_ms=13 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=3 mint=0 maxt=7200000 ulid=01GTWJWJK3AACKM9083E9S5JXJ sources="[01GTWJWC6NREBNJ5YWKA73NWEF 01GTWJW0819ZTCRZ2DD7YKKBG3 01GTWJW5EWHRYWGJ09A309FMT1]" duration=8.600083084s shard=1_of_2 level=info component=compactor msg="compact blocks" count=3 mint=0 maxt=7200000 ulid=01GTWJWJK34P0WH5Z2NXJBHEPQ sources="[01GTWJWC6NREBNJ5YWKA73NWEF 01GTWJW0819ZTCRZ2DD7YKKBG3 01GTWJW5EWHRYWGJ09A309FMT1]" duration=8.600455084s shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compacted blocks" new="[01GTWJWJK3AACKM9083E9S5JXJ 01GTWJWJK34P0WH5Z2NXJBHEPQ]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3469273953/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJW0819ZTCRZ2DD7YKKBG3 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3469273953/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJW5EWHRYWGJ09A309FMT1 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3469273953/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJWC6NREBNJ5YWKA73NWEF]" duration=8.601002084s duration_ms=8601 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3469273953/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJWJK34P0WH5Z2NXJBHEPQ/chunks/000001 dst=01GTWJWJK34P0WH5Z2NXJBHEPQ/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3469273953/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3469273953/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJWJK34P0WH5Z2NXJBHEPQ/index dst=01GTWJWJK34P0WH5Z2NXJBHEPQ/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3469273953/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3469273953/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJWJK3AACKM9083E9S5JXJ/chunks/000001 dst=01GTWJWJK3AACKM9083E9S5JXJ/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3469273953/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded block" result_block=01GTWJWJK34P0WH5Z2NXJBHEPQ duration=7.194998ms duration_ms=7 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3469273953/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJWJK3AACKM9083E9S5JXJ/index dst=01GTWJWJK3AACKM9083E9S5JXJ/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3469273953/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded block" result_block=01GTWJWJK3AACKM9083E9S5JXJ duration=4.298999ms duration_ms=4 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=10.326998ms duration_ms=10 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GTWJW0819ZTCRZ2DD7YKKBG3 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GTWJW0819ZTCRZ2DD7YKKBG3 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GTWJW5EWHRYWGJ09A309FMT1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GTWJW5EWHRYWGJ09A309FMT1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GTWJWC6NREBNJ5YWKA73NWEF level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GTWJWC6NREBNJ5YWKA73NWEF level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compaction job succeeded" duration=8.632025077s duration_ms=8632 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=7.865998ms duration_ms=7 cached=5 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=8.278998ms duration_ms=8 cached=5 returned=2 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/splitting_should_be_disabled_if_configured_shards_=_0 level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=root3 ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=cleaner run_id=1678144086 msg="started blocks cleanup and maintenance" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=6.869998ms level=info component=cleaner run_id=1678144086 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=4.588999ms duration_ms=4 cached=3 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge--0-7200000 job="stage: merge, range start: 0, range end: 7200000, shard: , blocks: 01GTWJWV6W7JDXS80VPF7TD53E (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01GTWJX0SWHKH9XBYWVC1WK2AZ (min time: 1970-01-01 00:05:00 +0000 UTC, max time: 1970-01-01 00:10:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GTWJWV6W7JDXS80VPF7TD53E (min time: 0, max time: 300000) 01GTWJX0SWHKH9XBYWVC1WK2AZ (min time: 300000, max time: 600000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GTWJX0SWHKH9XBYWVC1WK2AZ/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GTWJWV6W7JDXS80VPF7TD53E/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_01639767630/001/compact/0@17241709254077376921-merge--0-7200000/01GTWJWV6W7JDXS80VPF7TD53E /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_01639767630/001/compact/0@17241709254077376921-merge--0-7200000/01GTWJX0SWHKH9XBYWVC1WK2AZ]" duration=6.900999ms duration_ms=6 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=600000 ulid=01GTWJX8J9H9MSHJGQP3S9DC8C sources="[01GTWJX0SWHKH9XBYWVC1WK2AZ 01GTWJWV6W7JDXS80VPF7TD53E]" duration=4.855284918s shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compacted blocks" new=[01GTWJX8J9H9MSHJGQP3S9DC8C] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_01639767630/001/compact/0@17241709254077376921-merge--0-7200000/01GTWJWV6W7JDXS80VPF7TD53E /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_01639767630/001/compact/0@17241709254077376921-merge--0-7200000/01GTWJX0SWHKH9XBYWVC1WK2AZ]" duration=4.855810917s duration_ms=4855 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_01639767630/001/compact/0@17241709254077376921-merge--0-7200000/01GTWJX8J9H9MSHJGQP3S9DC8C/chunks/000001" dst=01GTWJX8J9H9MSHJGQP3S9DC8C/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_01639767630/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_01639767630/001/compact/0@17241709254077376921-merge--0-7200000/01GTWJX8J9H9MSHJGQP3S9DC8C/index" dst=01GTWJX8J9H9MSHJGQP3S9DC8C/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_01639767630/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded block" result_block=01GTWJX8J9H9MSHJGQP3S9DC8C duration=17.960996ms duration_ms=17 external_labels={} level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=20.782996ms duration_ms=20 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GTWJWV6W7JDXS80VPF7TD53E level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01GTWJWV6W7JDXS80VPF7TD53E level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GTWJX0SWHKH9XBYWVC1WK2AZ level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01GTWJX0SWHKH9XBYWVC1WK2AZ level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction job succeeded" duration=4.888542911s duration_ms=4888 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=5.504999ms duration_ms=5 cached=4 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=9.817998ms duration_ms=9 cached=4 returned=2 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/non-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split level=info component=compactor msg="instance not found in the ring" instance=root3 ring=compactor level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=cleaner run_id=1678144108 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=35.447992ms duration_ms=35 cached=3 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job="stage: split, range start: 0, range end: 7200000, shard: 1_of_1, blocks: 01GTWJXDHG0DSFH8HT9F4SNH3T (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01GTWJXKC4VAPX59J2RSSKFFC6 (min time: 1970-01-01 00:05:00 +0000 UTC, max time: 1970-01-01 00:10:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GTWJXDHG0DSFH8HT9F4SNH3T (min time: 0, max time: 300000) 01GTWJXKC4VAPX59J2RSSKFFC6 (min time: 300000, max time: 600000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GTWJXDHG0DSFH8HT9F4SNH3T/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GTWJXKC4VAPX59J2RSSKFFC6/meta.json level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=58.839987ms level=info component=cleaner run_id=1678144108 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split1153461233/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJXDHG0DSFH8HT9F4SNH3T /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split1153461233/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJXKC4VAPX59J2RSSKFFC6]" duration=13.149997ms duration_ms=13 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=600000 ulid=01GTWJXY0CAXPSJH1XAXJ38MHS sources="[01GTWJXKC4VAPX59J2RSSKFFC6 01GTWJXDHG0DSFH8HT9F4SNH3T]" duration=10.190402729s shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=600000 ulid=01GTWJXY0C49FJS1Y6YMEZJYVX sources="[01GTWJXKC4VAPX59J2RSSKFFC6 01GTWJXDHG0DSFH8HT9F4SNH3T]" duration=10.190670729s shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compacted blocks" new="[01GTWJXY0CAXPSJH1XAXJ38MHS 01GTWJXY0C49FJS1Y6YMEZJYVX]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split1153461233/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJXDHG0DSFH8HT9F4SNH3T /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split1153461233/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJXKC4VAPX59J2RSSKFFC6]" duration=10.191027729s duration_ms=10191 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split1153461233/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJXY0C49FJS1Y6YMEZJYVX/chunks/000001 dst=01GTWJXY0C49FJS1Y6YMEZJYVX/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split1153461233/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split1153461233/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJXY0CAXPSJH1XAXJ38MHS/chunks/000001 dst=01GTWJXY0CAXPSJH1XAXJ38MHS/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split1153461233/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split1153461233/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJXY0C49FJS1Y6YMEZJYVX/index dst=01GTWJXY0C49FJS1Y6YMEZJYVX/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split1153461233/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split1153461233/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJXY0CAXPSJH1XAXJ38MHS/index dst=01GTWJXY0CAXPSJH1XAXJ38MHS/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split1153461233/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded block" result_block=01GTWJXY0C49FJS1Y6YMEZJYVX duration=3.430999ms duration_ms=3 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded block" result_block=01GTWJXY0CAXPSJH1XAXJ38MHS duration=3.445999ms duration_ms=3 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=6.423998ms duration_ms=6 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GTWJXDHG0DSFH8HT9F4SNH3T level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01GTWJXDHG0DSFH8HT9F4SNH3T level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GTWJXKC4VAPX59J2RSSKFFC6 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01GTWJXKC4VAPX59J2RSSKFFC6 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction job succeeded" duration=10.216245723s duration_ms=10216 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=6.504998ms duration_ms=6 cached=5 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=8.888998ms duration_ms=8 cached=5 returned=3 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/non-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=root3 ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=cleaner run_id=1678144135 msg="started blocks cleanup and maintenance" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=10.143998ms duration_ms=10 cached=3 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job="stage: split, range start: 0, range end: 7200000, shard: 1_of_1, blocks: 01GTWJY86GPPK96H6HBRD2YTAE (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01GTWJYD225QBPTMJMF770NATG (min time: 1970-01-01 00:07:00 +0000 UTC, max time: 1970-01-01 00:10:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GTWJY86GPPK96H6HBRD2YTAE (min time: 0, max time: 300000) 01GTWJYD225QBPTMJMF770NATG (min time: 420000, max time: 600000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GTWJYD225QBPTMJMF770NATG/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GTWJY86GPPK96H6HBRD2YTAE/meta.json level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=23.602995ms level=info component=cleaner run_id=1678144135 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split367476009/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJY86GPPK96H6HBRD2YTAE /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split367476009/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJYD225QBPTMJMF770NATG]" duration=21.986996ms duration_ms=21 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=600000 ulid=01GTWJYQZ0FYREEAQJ7CR3QH6Y sources="[01GTWJYD225QBPTMJMF770NATG 01GTWJY86GPPK96H6HBRD2YTAE]" duration=8.180089178s shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=600000 ulid=01GTWJYQZ090MJ6SRAZFA8KTX8 sources="[01GTWJYD225QBPTMJMF770NATG 01GTWJY86GPPK96H6HBRD2YTAE]" duration=8.180296178s shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compacted blocks" new="[01GTWJYQZ0FYREEAQJ7CR3QH6Y 01GTWJYQZ090MJ6SRAZFA8KTX8]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split367476009/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJY86GPPK96H6HBRD2YTAE /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split367476009/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJYD225QBPTMJMF770NATG]" duration=8.180647178s duration_ms=8180 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split367476009/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJYQZ090MJ6SRAZFA8KTX8/chunks/000001 dst=01GTWJYQZ090MJ6SRAZFA8KTX8/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split367476009/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split367476009/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJYQZ0FYREEAQJ7CR3QH6Y/chunks/000001 dst=01GTWJYQZ0FYREEAQJ7CR3QH6Y/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split367476009/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split367476009/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJYQZ0FYREEAQJ7CR3QH6Y/index dst=01GTWJYQZ0FYREEAQJ7CR3QH6Y/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split367476009/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split367476009/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJYQZ090MJ6SRAZFA8KTX8/index dst=01GTWJYQZ090MJ6SRAZFA8KTX8/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split367476009/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded block" result_block=01GTWJYQZ090MJ6SRAZFA8KTX8 duration=7.596998ms duration_ms=7 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded block" result_block=01GTWJYQZ0FYREEAQJ7CR3QH6Y duration=6.976999ms duration_ms=6 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=13.271997ms duration_ms=13 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GTWJY86GPPK96H6HBRD2YTAE level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01GTWJY86GPPK96H6HBRD2YTAE level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GTWJYD225QBPTMJMF770NATG level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01GTWJYD225QBPTMJMF770NATG level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction job succeeded" duration=8.233465165s duration_ms=8233 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=8.227999ms duration_ms=8 cached=5 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=14.944996ms duration_ms=14 cached=5 returned=3 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=root3 ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=cleaner run_id=1678144155 msg="started blocks cleanup and maintenance" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=8.091998ms duration_ms=8 cached=3 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=7.156999ms level=info component=cleaner run_id=1678144155 msg="successfully completed blocks cleanup and maintenance" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job="stage: split, range start: 0, range end: 7200000, shard: 1_of_1, blocks: 01GTWJZ09E3QDC0JMGH9EFQPVM (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01GTWJZ595QK8DR7WVG1EMR2XT (min time: 1970-01-01 00:01:00 +0000 UTC, max time: 1970-01-01 00:07:00 +0000 UTC),01GTWJZ8Q739TT9NZ8R3DC58E2 (min time: 1970-01-01 01:00:00 +0000 UTC, max time: 1970-01-01 02:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=3 plan="[01GTWJZ09E3QDC0JMGH9EFQPVM (min time: 0, max time: 300000) 01GTWJZ595QK8DR7WVG1EMR2XT (min time: 60000, max time: 420000) 01GTWJZ8Q739TT9NZ8R3DC58E2 (min time: 3600000, max time: 7200000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GTWJZ09E3QDC0JMGH9EFQPVM/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GTWJZ8Q739TT9NZ8R3DC58E2/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GTWJZ595QK8DR7WVG1EMR2XT/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=3 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together2406240653/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJZ09E3QDC0JMGH9EFQPVM /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together2406240653/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJZ595QK8DR7WVG1EMR2XT /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together2406240653/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJZ8Q739TT9NZ8R3DC58E2]" duration=15.753997ms duration_ms=15 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=3 mint=0 maxt=7200000 ulid=01GTWJZC1YK2QVBQ6YC8NTZ0TT sources="[01GTWJZ09E3QDC0JMGH9EFQPVM 01GTWJZ595QK8DR7WVG1EMR2XT 01GTWJZ8Q739TT9NZ8R3DC58E2]" duration=6.161658627s shard=1_of_2 level=info component=compactor msg="compact blocks" count=3 mint=0 maxt=7200000 ulid=01GTWJZC1YJAM0SVP9PXHGT9QE sources="[01GTWJZ09E3QDC0JMGH9EFQPVM 01GTWJZ595QK8DR7WVG1EMR2XT 01GTWJZ8Q739TT9NZ8R3DC58E2]" duration=6.162098627s shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compacted blocks" new="[01GTWJZC1YK2QVBQ6YC8NTZ0TT 01GTWJZC1YJAM0SVP9PXHGT9QE]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together2406240653/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJZ09E3QDC0JMGH9EFQPVM /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together2406240653/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJZ595QK8DR7WVG1EMR2XT /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together2406240653/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJZ8Q739TT9NZ8R3DC58E2]" duration=6.162834626s duration_ms=6162 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together2406240653/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJZC1YJAM0SVP9PXHGT9QE/chunks/000001 dst=01GTWJZC1YJAM0SVP9PXHGT9QE/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together2406240653/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together2406240653/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJZC1YJAM0SVP9PXHGT9QE/index dst=01GTWJZC1YJAM0SVP9PXHGT9QE/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together2406240653/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded block" result_block=01GTWJZC1YJAM0SVP9PXHGT9QE duration=3.711999ms duration_ms=3 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together2406240653/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJZC1YK2QVBQ6YC8NTZ0TT/chunks/000001 dst=01GTWJZC1YK2QVBQ6YC8NTZ0TT/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together2406240653/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together2406240653/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GTWJZC1YK2QVBQ6YC8NTZ0TT/index dst=01GTWJZC1YK2QVBQ6YC8NTZ0TT/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together2406240653/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded block" result_block=01GTWJZC1YK2QVBQ6YC8NTZ0TT duration=6.927998ms duration_ms=6 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=10.415998ms duration_ms=10 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GTWJZ09E3QDC0JMGH9EFQPVM level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GTWJZ09E3QDC0JMGH9EFQPVM level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GTWJZ595QK8DR7WVG1EMR2XT level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GTWJZ595QK8DR7WVG1EMR2XT level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GTWJZ8Q739TT9NZ8R3DC58E2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GTWJZ8Q739TT9NZ8R3DC58E2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compaction job succeeded" duration=6.198032618s duration_ms=6198 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=9.474998ms duration_ms=9 cached=5 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=10.150998ms duration_ms=10 cached=5 returned=2 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/should_correctly_handle_empty_blocks_generated_in_the_splitting_stage level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=root3 ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1678144166 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=6.110999ms duration_ms=6 cached=1 returned=1 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job="stage: split, range start: 7200000, range end: 14400000, shard: 1_of_1, blocks: 01GTWJZJCZCDZCRM02BJBQE1CX (min time: 1970-01-01 03:59:59.999 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=1 plan="[01GTWJZJCZCDZCRM02BJBQE1CX (min time: 14399999, max time: 14400000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GTWJZJCZCDZCRM02BJBQE1CX/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=1 plan=[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorshould_correctly_handle_empty_blocks_generated_in_the_splitting_stage950071558/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GTWJZJCZCDZCRM02BJBQE1CX] duration=14.572996ms duration_ms=14 level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=22.645995ms level=info component=cleaner run_id=1678144166 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor msg="compact blocks" count=1 mint=14399999 maxt=14400000 ulid=01GTWJZPHD05Y09F1BRTQWR9WP sources=[01GTWJZJCZCDZCRM02BJBQE1CX] duration=5.236542833s shard=1_of_2 level=info component=compactor msg="compact blocks resulted in empty block" count=1 sources=[01GTWJZJCZCDZCRM02BJBQE1CX] duration=5.236729833s shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new="[01GTWJZPHD05Y09F1BRTQWR9WP 00000000000000000000000000]" blocks=[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorshould_correctly_handle_empty_blocks_generated_in_the_splitting_stage950071558/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GTWJZJCZCDZCRM02BJBQE1CX] duration=5.236948833s duration_ms=5236 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction produced an empty block" shard_id=2_of_2 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorshould_correctly_handle_empty_blocks_generated_in_the_splitting_stage950071558/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GTWJZPHD05Y09F1BRTQWR9WP/chunks/000001 dst=01GTWJZPHD05Y09F1BRTQWR9WP/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorshould_correctly_handle_empty_blocks_generated_in_the_splitting_stage950071558/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorshould_correctly_handle_empty_blocks_generated_in_the_splitting_stage950071558/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GTWJZPHD05Y09F1BRTQWR9WP/index dst=01GTWJZPHD05Y09F1BRTQWR9WP/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorshould_correctly_handle_empty_blocks_generated_in_the_splitting_stage950071558/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GTWJZPHD05Y09F1BRTQWR9WP duration=13.598997ms duration_ms=13 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=16.109997ms duration_ms=16 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GTWJZJCZCDZCRM02BJBQE1CX level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GTWJZJCZCDZCRM02BJBQE1CX level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=5.270659825s duration_ms=5270 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=3.677999ms duration_ms=3 cached=2 returned=1 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=4.692999ms duration_ms=4 cached=2 returned=1 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/splitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0 level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=root3 ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1678144208 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=20.478996ms duration_ms=20 cached=10 returned=10 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job="stage: merge, range start: 14400000, range end: 28800000, shard: 1_of_2, blocks: 01GTWK0A67JY2C574951QNR3VA (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 06:00:00 +0000 UTC),01GTWK0H93G6E4KMDD5G7BJKFV (min time: 1970-01-01 06:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job="stage: merge, range start: 14400000, range end: 28800000, shard: 2_of_2, blocks: 01GTWK0E9GBAHWTFVAWP7FX8P0 (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 06:00:00 +0000 UTC),01GTWK0MWB7ZC6MB5HQP3CREK1 (min time: 1970-01-01 06:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge--28800000-57600000 job="stage: merge, range start: 28800000, range end: 57600000, shard: , blocks: 01GTWK0RFB67ANFXP671WKGP8V (min time: 1970-01-01 08:00:00 +0000 UTC, max time: 1970-01-01 10:00:00 +0000 UTC),01GTWK0VKW5XYG77P04A15AP0Y (min time: 1970-01-01 14:00:00 +0000 UTC, max time: 1970-01-01 16:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge--7200000-14400000 job="stage: merge, range start: 7200000, range end: 14400000, shard: , blocks: 01GTWK039B3Q3FR8ZR4YJD46M6 (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC),01GTWK06P77ER4E3NK5REGCTX5 (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GTWK039B3Q3FR8ZR4YJD46M6 (min time: 7200000, max time: 14400000) 01GTWK06P77ER4E3NK5REGCTX5 (min time: 7200000, max time: 14400000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GTWK039B3Q3FR8ZR4YJD46M6/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GTWK06P77ER4E3NK5REGCTX5/meta.json level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=27.381994ms level=info component=cleaner run_id=1678144208 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_04081158433/001/compact/0@17241709254077376921-merge--7200000-14400000/01GTWK039B3Q3FR8ZR4YJD46M6 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_04081158433/001/compact/0@17241709254077376921-merge--7200000-14400000/01GTWK06P77ER4E3NK5REGCTX5]" duration=11.658998ms duration_ms=11 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01GTWK0Z6N9Y43G3TVHY2CRSVS sources="[01GTWK039B3Q3FR8ZR4YJD46M6 01GTWK06P77ER4E3NK5REGCTX5]" duration=3.132892302s shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new=[01GTWK0Z6N9Y43G3TVHY2CRSVS] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_04081158433/001/compact/0@17241709254077376921-merge--7200000-14400000/01GTWK039B3Q3FR8ZR4YJD46M6 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_04081158433/001/compact/0@17241709254077376921-merge--7200000-14400000/01GTWK06P77ER4E3NK5REGCTX5]" duration=3.133439302s duration_ms=3133 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_04081158433/001/compact/0@17241709254077376921-merge--7200000-14400000/01GTWK0Z6N9Y43G3TVHY2CRSVS/chunks/000001" dst=01GTWK0Z6N9Y43G3TVHY2CRSVS/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_04081158433/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_04081158433/001/compact/0@17241709254077376921-merge--7200000-14400000/01GTWK0Z6N9Y43G3TVHY2CRSVS/index" dst=01GTWK0Z6N9Y43G3TVHY2CRSVS/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_04081158433/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GTWK0Z6N9Y43G3TVHY2CRSVS duration=3.499999ms duration_ms=3 external_labels={} level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=6.269999ms duration_ms=6 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GTWK039B3Q3FR8ZR4YJD46M6 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GTWK039B3Q3FR8ZR4YJD46M6 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GTWK06P77ER4E3NK5REGCTX5 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GTWK06P77ER4E3NK5REGCTX5 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=3.156268297s duration_ms=3156 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GTWK0A67JY2C574951QNR3VA (min time: 14400000, max time: 21600000) 01GTWK0H93G6E4KMDD5G7BJKFV (min time: 21600000, max time: 28800000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GTWK0A67JY2C574951QNR3VA/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GTWK0H93G6E4KMDD5G7BJKFV/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_04081158433/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GTWK0A67JY2C574951QNR3VA /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_04081158433/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GTWK0H93G6E4KMDD5G7BJKFV]" duration=5.905998ms duration_ms=5 level=info component=compactor msg="compact blocks" count=2 mint=14400000 maxt=28800000 ulid=01GTWK12950MQS05K27HVGF4BN sources="[01GTWK0A67JY2C574951QNR3VA 01GTWK0H93G6E4KMDD5G7BJKFV]" duration=4.512812995s shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compacted blocks" new=[01GTWK12950MQS05K27HVGF4BN] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_04081158433/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GTWK0A67JY2C574951QNR3VA /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_04081158433/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GTWK0H93G6E4KMDD5G7BJKFV]" duration=4.513229995s duration_ms=4513 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_04081158433/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GTWK12950MQS05K27HVGF4BN/chunks/000001" dst=01GTWK12950MQS05K27HVGF4BN/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_04081158433/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_04081158433/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GTWK12950MQS05K27HVGF4BN/index" dst=01GTWK12950MQS05K27HVGF4BN/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_04081158433/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded block" result_block=01GTWK12950MQS05K27HVGF4BN duration=3.545999ms duration_ms=3 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=6.586999ms duration_ms=6 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GTWK0A67JY2C574951QNR3VA level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GTWK0A67JY2C574951QNR3VA level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GTWK0H93G6E4KMDD5G7BJKFV level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GTWK0H93G6E4KMDD5G7BJKFV level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction job succeeded" duration=4.53029199s duration_ms=4530 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GTWK0E9GBAHWTFVAWP7FX8P0 (min time: 14400000, max time: 21600000) 01GTWK0MWB7ZC6MB5HQP3CREK1 (min time: 21600000, max time: 28800000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GTWK0E9GBAHWTFVAWP7FX8P0/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GTWK0MWB7ZC6MB5HQP3CREK1/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_04081158433/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GTWK0E9GBAHWTFVAWP7FX8P0 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_04081158433/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GTWK0MWB7ZC6MB5HQP3CREK1]" duration=9.628998ms duration_ms=9 level=info component=compactor msg="compact blocks" count=2 mint=14400000 maxt=28800000 ulid=01GTWK16PV42KY9PE028ME4PMN sources="[01GTWK0E9GBAHWTFVAWP7FX8P0 01GTWK0MWB7ZC6MB5HQP3CREK1]" duration=4.336290033s shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compacted blocks" new=[01GTWK16PV42KY9PE028ME4PMN] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_04081158433/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GTWK0E9GBAHWTFVAWP7FX8P0 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_04081158433/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GTWK0MWB7ZC6MB5HQP3CREK1]" duration=4.336990033s duration_ms=4336 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_04081158433/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GTWK16PV42KY9PE028ME4PMN/chunks/000001" dst=01GTWK16PV42KY9PE028ME4PMN/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_04081158433/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_04081158433/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GTWK16PV42KY9PE028ME4PMN/index" dst=01GTWK16PV42KY9PE028ME4PMN/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_04081158433/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded block" result_block=01GTWK16PV42KY9PE028ME4PMN duration=4.285999ms duration_ms=4 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=7.343999ms duration_ms=7 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GTWK0E9GBAHWTFVAWP7FX8P0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GTWK0E9GBAHWTFVAWP7FX8P0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GTWK0MWB7ZC6MB5HQP3CREK1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GTWK0MWB7ZC6MB5HQP3CREK1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction job succeeded" duration=4.358593029s duration_ms=4358 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GTWK0RFB67ANFXP671WKGP8V (min time: 28800000, max time: 36000000) 01GTWK0VKW5XYG77P04A15AP0Y (min time: 50400000, max time: 57600000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GTWK0VKW5XYG77P04A15AP0Y/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GTWK0RFB67ANFXP671WKGP8V/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_04081158433/001/compact/0@17241709254077376921-merge--28800000-57600000/01GTWK0RFB67ANFXP671WKGP8V /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_04081158433/001/compact/0@17241709254077376921-merge--28800000-57600000/01GTWK0VKW5XYG77P04A15AP0Y]" duration=6.321998ms duration_ms=6 split_merge_compactor_test.go:640: expected , got Diff: --- metric output does not match expectation; want +++ got: @@ -2,3 +2,3 @@ # TYPE cortex_compactor_runs_completed_total counter -cortex_compactor_runs_completed_total 1 +cortex_compactor_runs_completed_total 0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor level=error component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compaction job failed" duration=5.730327723s duration_ms=5730 err="compact blocks [/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_04081158433/001/compact/0@17241709254077376921-merge--28800000-57600000/01GTWK0RFB67ANFXP671WKGP8V /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_04081158433/001/compact/0@17241709254077376921-merge--28800000-57600000/01GTWK0VKW5XYG77P04A15AP0Y]: 2 errors: populate block: adding series: asyncBlockWriter doesn't run anymore: add series: context canceled; context canceled" level=error component=compactor msg="failed to compact user blocks" user=user-1 err="compaction: group 0@17241709254077376921-merge--28800000-57600000: compact blocks [/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_04081158433/001/compact/0@17241709254077376921-merge--28800000-57600000/01GTWK0RFB67ANFXP671WKGP8V /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_04081158433/001/compact/0@17241709254077376921-merge--28800000-57600000/01GTWK0VKW5XYG77P04A15AP0Y]: 2 errors: populate block: adding series: asyncBlockWriter doesn't run anymore: add series: context canceled; context canceled" --- FAIL: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor (310.51s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split (18.76s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split (15.05s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split (20.31s) --- FAIL: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/smaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges (75.68s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together (27.58s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/splitting_should_be_disabled_if_configured_shards_=_0 (18.80s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/non-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split (27.27s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/non-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split (24.69s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together (18.52s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/should_correctly_handle_empty_blocks_generated_in_the_splitting_stage (9.75s) --- FAIL: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/splitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0 (54.08s) === RUN TestPlanCompaction === RUN TestPlanCompaction/should_split_a_single_block_if_<_smallest_compaction_range split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 20, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_within_the_same_time_range split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/should_not_split_non-split_blocks_if_they're_>_smallest_compaction_range_(do_not_split_historical_blocks_after_enabling_splitting) split_merge_grouper_test.go:515: got 2 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 0, range end: 20, shard: 1_of_2, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: merge, range start: 0, range end: 20, shard: 2_of_2, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/should_not_merge_blocks_within_the_same_time_range_but_with_different_external_labels split_merge_grouper_test.go:515: got 3 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_1, blocks: 00000000020000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC),00000000030000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_1, blocks: 00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/a_block_with_time_range_larger_then_the_largest_compaction_range_should_NOT_be_considered_for_compaction split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 80, range end: 120, shard: 1_of_1, blocks: 00000000040000000000000000 (min time: 1970-01-01 00:00:00.08 +0000 UTC, max time: 1970-01-01 00:00:00.12 +0000 UTC),00000000050000000000000000 (min time: 1970-01-01 00:00:00.08 +0000 UTC, max time: 1970-01-01 00:00:00.12 +0000 UTC) === RUN TestPlanCompaction/should_NOT_split_a_single_block_if_==_smallest_compaction_range_but_configured_shards_=_0 split_merge_grouper_test.go:515: got 0 jobs: === RUN TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_in_different_time_ranges,_two_split_groups split_merge_grouper_test.go:515: got 4 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 10, shard: 1_of_2, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 10, shard: 2_of_2, blocks: 00000000020000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_2, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 2_of_2, blocks: 00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/should_merge_but_NOT_split_multiple_1st_level_blocks_in_different_time_ranges_if_configured_shards_=_0 split_merge_grouper_test.go:515: got 2 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 0, range end: 10, shard: , blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC) split_merge_grouper_test.go:517: - stage: merge, range start: 10, range end: 20, shard: , blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/input_blocks_can_be_compacted_on_a_mix_of_1st_and_2nd_ranges,_guaranteeing_no_overlaps_and_giving_preference_to_smaller_ranges split_merge_grouper_test.go:515: got 3 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 20, range end: 40, shard: 1_of_1, blocks: 00000000040000000000000000 (min time: 1970-01-01 00:00:00.02 +0000 UTC, max time: 1970-01-01 00:00:00.03 +0000 UTC),00000000050000000000000000 (min time: 1970-01-01 00:00:00.03 +0000 UTC, max time: 1970-01-01 00:00:00.04 +0000 UTC) split_merge_grouper_test.go:517: - stage: merge, range start: 70, range end: 80, shard: 1_of_1, blocks: 00000000080000000000000000 (min time: 1970-01-01 00:00:00.07 +0000 UTC, max time: 1970-01-01 00:00:00.08 +0000 UTC),00000000090000000000000000 (min time: 1970-01-01 00:00:00.075 +0000 UTC, max time: 1970-01-01 00:00:00.08 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 10, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00.007 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC) === RUN TestPlanCompaction/a_range_containing_the_most_recent_block_shouldn't_be_prematurely_compacted_if_doesn't_cover_the_full_range split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 10, shard: 1_of_1, blocks: 00000000000000000000000000 (min time: 1970-01-01 00:00:00.005 +0000 UTC, max time: 1970-01-01 00:00:00.008 +0000 UTC),00000000000000000000000000 (min time: 1970-01-01 00:00:00.007 +0000 UTC, max time: 1970-01-01 00:00:00.009 +0000 UTC) === RUN TestPlanCompaction/input_blocks_match_the_largest_range_but_can_be_compacted_because_overlapping split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 80, range end: 120, shard: 1_of_1, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00.08 +0000 UTC, max time: 1970-01-01 00:00:00.12 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.08 +0000 UTC, max time: 1970-01-01 00:00:00.12 +0000 UTC) === RUN TestPlanCompaction/a_block_with_time_range_crossing_two_1st_level_ranges_should_be_NOT_considered_for_1st_level_splitting split_merge_grouper_test.go:515: got 2 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 20, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 20, range end: 40, shard: 1_of_1, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00.02 +0000 UTC, max time: 1970-01-01 00:00:00.03 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.03 +0000 UTC, max time: 1970-01-01 00:00:00.04 +0000 UTC) === RUN TestPlanCompaction/no_input_blocks split_merge_grouper_test.go:515: got 0 jobs: === RUN TestPlanCompaction/should_split_a_single_block_if_==_smallest_compaction_range split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 20, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_in_different_time_ranges split_merge_grouper_test.go:515: got 2 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 10, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_1, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_in_different_time_ranges,_single_split_group split_merge_grouper_test.go:515: got 2 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 10, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_1, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/should_merge_split_blocks_that_can_be_compacted_on_the_2nd_range_only split_merge_grouper_test.go:515: got 3 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 0, range end: 20, shard: 1_of_2, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: merge, range start: 0, range end: 20, shard: 2_of_2, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: merge, range start: 20, range end: 40, shard: 1_of_2, blocks: 00000000050000000000000000 (min time: 1970-01-01 00:00:00.02 +0000 UTC, max time: 1970-01-01 00:00:00.03 +0000 UTC),00000000060000000000000000 (min time: 1970-01-01 00:00:00.03 +0000 UTC, max time: 1970-01-01 00:00:00.04 +0000 UTC) === RUN TestPlanCompaction/input_blocks_have_already_been_compacted_with_the_largest_range split_merge_grouper_test.go:515: got 0 jobs: === RUN TestPlanCompaction/a_block_with_time_range_crossing_two_1st_level_ranges_should_BE_considered_for_2nd_level_compaction split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 0, range end: 40, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.03 +0000 UTC),00000000030000000000000000 (min time: 1970-01-01 00:00:00.02 +0000 UTC, max time: 1970-01-01 00:00:00.04 +0000 UTC) --- PASS: TestPlanCompaction (0.03s) --- PASS: TestPlanCompaction/should_split_a_single_block_if_<_smallest_compaction_range (0.00s) --- PASS: TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_within_the_same_time_range (0.00s) --- PASS: TestPlanCompaction/should_not_split_non-split_blocks_if_they're_>_smallest_compaction_range_(do_not_split_historical_blocks_after_enabling_splitting) (0.00s) --- PASS: TestPlanCompaction/should_not_merge_blocks_within_the_same_time_range_but_with_different_external_labels (0.01s) --- PASS: TestPlanCompaction/a_block_with_time_range_larger_then_the_largest_compaction_range_should_NOT_be_considered_for_compaction (0.00s) --- PASS: TestPlanCompaction/should_NOT_split_a_single_block_if_==_smallest_compaction_range_but_configured_shards_=_0 (0.00s) --- PASS: TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_in_different_time_ranges,_two_split_groups (0.00s) --- PASS: TestPlanCompaction/should_merge_but_NOT_split_multiple_1st_level_blocks_in_different_time_ranges_if_configured_shards_=_0 (0.00s) --- PASS: TestPlanCompaction/input_blocks_can_be_compacted_on_a_mix_of_1st_and_2nd_ranges,_guaranteeing_no_overlaps_and_giving_preference_to_smaller_ranges (0.00s) --- PASS: TestPlanCompaction/a_range_containing_the_most_recent_block_shouldn't_be_prematurely_compacted_if_doesn't_cover_the_full_range (0.00s) --- PASS: TestPlanCompaction/input_blocks_match_the_largest_range_but_can_be_compacted_because_overlapping (0.00s) --- PASS: TestPlanCompaction/a_block_with_time_range_crossing_two_1st_level_ranges_should_be_NOT_considered_for_1st_level_splitting (0.00s) --- PASS: TestPlanCompaction/no_input_blocks (0.00s) --- PASS: TestPlanCompaction/should_split_a_single_block_if_==_smallest_compaction_range (0.00s) --- PASS: TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_in_different_time_ranges (0.00s) --- PASS: TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_in_different_time_ranges,_single_split_group (0.00s) --- PASS: TestPlanCompaction/should_merge_split_blocks_that_can_be_compacted_on_the_2nd_range_only (0.00s) --- PASS: TestPlanCompaction/input_blocks_have_already_been_compacted_with_the_largest_range (0.00s) --- PASS: TestPlanCompaction/a_block_with_time_range_crossing_two_1st_level_ranges_should_BE_considered_for_2nd_level_compaction (0.00s) === RUN TestPlanSplitting === RUN TestPlanSplitting/should_splitGroups_split_jobs_if_the_input_group_contains_multiple_non-sharded_blocks === RUN TestPlanSplitting/should_return_nil_if_the_input_group_is_empty === RUN TestPlanSplitting/should_return_nil_if_the_input_group_contains_no_non-sharded_blocks === RUN TestPlanSplitting/should_return_a_split_job_if_the_input_group_contains_1_non-sharded_block --- PASS: TestPlanSplitting (0.00s) --- PASS: TestPlanSplitting/should_splitGroups_split_jobs_if_the_input_group_contains_multiple_non-sharded_blocks (0.00s) --- PASS: TestPlanSplitting/should_return_nil_if_the_input_group_is_empty (0.00s) --- PASS: TestPlanSplitting/should_return_nil_if_the_input_group_contains_no_non-sharded_blocks (0.00s) --- PASS: TestPlanSplitting/should_return_a_split_job_if_the_input_group_contains_1_non-sharded_block (0.00s) === RUN TestGroupBlocksByShardID === RUN TestGroupBlocksByShardID/no_input_blocks === RUN TestGroupBlocksByShardID/only_1_block_in_input_with_shard_ID === RUN TestGroupBlocksByShardID/only_1_block_in_input_without_shard_ID === RUN TestGroupBlocksByShardID/multiple_blocks_per_shard_ID --- PASS: TestGroupBlocksByShardID (0.01s) --- PASS: TestGroupBlocksByShardID/no_input_blocks (0.00s) --- PASS: TestGroupBlocksByShardID/only_1_block_in_input_with_shard_ID (0.00s) --- PASS: TestGroupBlocksByShardID/only_1_block_in_input_without_shard_ID (0.00s) --- PASS: TestGroupBlocksByShardID/multiple_blocks_per_shard_ID (0.01s) === RUN TestGroupBlocksByRange === RUN TestGroupBlocksByRange/only_1_block_per_range === RUN TestGroupBlocksByRange/multiple_blocks_per_range === RUN TestGroupBlocksByRange/a_block_with_time_range_larger_then_the_range_should_be_excluded === RUN TestGroupBlocksByRange/blocks_with_different_time_ranges_but_all_fitting_within_the_input_range === RUN TestGroupBlocksByRange/no_input_blocks === RUN TestGroupBlocksByRange/only_1_block_in_input --- PASS: TestGroupBlocksByRange (0.00s) --- PASS: TestGroupBlocksByRange/only_1_block_per_range (0.00s) --- PASS: TestGroupBlocksByRange/multiple_blocks_per_range (0.00s) --- PASS: TestGroupBlocksByRange/a_block_with_time_range_larger_then_the_range_should_be_excluded (0.00s) --- PASS: TestGroupBlocksByRange/blocks_with_different_time_ranges_but_all_fitting_within_the_input_range (0.00s) --- PASS: TestGroupBlocksByRange/no_input_blocks (0.00s) --- PASS: TestGroupBlocksByRange/only_1_block_in_input (0.00s) === RUN TestJob_conflicts === RUN TestJob_conflicts/should_NOT_conflict_if_jobs_compact_different_blocks_with_non-overlapping_time_ranges_and_same_shard === RUN TestJob_conflicts/should_NOT_conflict_if_jobs_compact_same_blocks_with_overlapping_time_ranges_but_different_shard === RUN TestJob_conflicts/should_conflict_if_jobs_compact_same_blocks_with_overlapping_time_ranges_and_different_shard_but_at_a_different_stage === RUN TestJob_conflicts/should_conflict_between_split_and_merge_jobs_with_overlapping_time_ranges === RUN TestJob_conflicts/should_NOT_conflict_between_split_and_merge_jobs_with_non-overlapping_time_ranges === RUN TestJob_conflicts/should_conflict_if_jobs_compact_different_blocks_but_with_overlapping_time_ranges_and_same_shard --- PASS: TestJob_conflicts (0.01s) --- PASS: TestJob_conflicts/should_NOT_conflict_if_jobs_compact_different_blocks_with_non-overlapping_time_ranges_and_same_shard (0.00s) --- PASS: TestJob_conflicts/should_NOT_conflict_if_jobs_compact_same_blocks_with_overlapping_time_ranges_but_different_shard (0.00s) --- PASS: TestJob_conflicts/should_conflict_if_jobs_compact_same_blocks_with_overlapping_time_ranges_and_different_shard_but_at_a_different_stage (0.00s) --- PASS: TestJob_conflicts/should_conflict_between_split_and_merge_jobs_with_overlapping_time_ranges (0.00s) --- PASS: TestJob_conflicts/should_NOT_conflict_between_split_and_merge_jobs_with_non-overlapping_time_ranges (0.00s) --- PASS: TestJob_conflicts/should_conflict_if_jobs_compact_different_blocks_but_with_overlapping_time_ranges_and_same_shard (0.00s) === RUN TestBlocksGroup_overlaps --- PASS: TestBlocksGroup_overlaps (0.00s) === RUN TestBlocksGroup_getNonShardedBlocks --- PASS: TestBlocksGroup_getNonShardedBlocks (0.00s) === RUN TestSplitAndMergePlanner_Plan === RUN TestSplitAndMergePlanner_Plan/no_blocks === RUN TestSplitAndMergePlanner_Plan/a_source_block_is_larger_then_the_largest_range === RUN TestSplitAndMergePlanner_Plan/source_blocks_are_smaller_then_the_largest_range_but_compacted_block_is_larger === RUN TestSplitAndMergePlanner_Plan/source_blocks_and_compacted_block_are_smaller_then_the_largest_range_but_misaligned === RUN TestSplitAndMergePlanner_Plan/blocks_fit_within_the_largest_range --- PASS: TestSplitAndMergePlanner_Plan (0.00s) --- PASS: TestSplitAndMergePlanner_Plan/no_blocks (0.00s) --- PASS: TestSplitAndMergePlanner_Plan/a_source_block_is_larger_then_the_largest_range (0.00s) --- PASS: TestSplitAndMergePlanner_Plan/source_blocks_are_smaller_then_the_largest_range_but_compacted_block_is_larger (0.00s) --- PASS: TestSplitAndMergePlanner_Plan/source_blocks_and_compacted_block_are_smaller_then_the_largest_range_but_misaligned (0.00s) --- PASS: TestSplitAndMergePlanner_Plan/blocks_fit_within_the_largest_range (0.00s) === RUN TestSyncerMetrics --- PASS: TestSyncerMetrics (0.01s) === RUN TestDeleteTenant --- PASS: TestDeleteTenant (0.15s) === RUN TestDeleteTenantStatus === RUN TestDeleteTenantStatus/empty === RUN TestDeleteTenantStatus/no_user_objects === RUN TestDeleteTenantStatus/non-block_files === RUN TestDeleteTenantStatus/block_files --- PASS: TestDeleteTenantStatus (0.58s) --- PASS: TestDeleteTenantStatus/empty (0.16s) --- PASS: TestDeleteTenantStatus/no_user_objects (0.17s) --- PASS: TestDeleteTenantStatus/non-block_files (0.06s) --- PASS: TestDeleteTenantStatus/block_files (0.19s) === CONT TestSyncer_GarbageCollect_e2e === CONT TestMultitenantCompactor_ShouldFailCompactionOnTimeout === RUN TestSyncer_GarbageCollect_e2e/inmem create 00000000000000000000000000 create 00000000010000000000000000 create 00000000020000000000000000 create 00000000030000000000000000 create 00000000040000000000000000 create 00000000050000000000000000 create 00000000060000000000000000 create 00000000070000000000000000 create 00000000080000000000000000 create 00000000090000000000000000 create 00000000340000000000000000 create 00000000680000000000000000 create 000000009C0000000000000000 create 00000000CG0000000000000000 create 00000000FM0000000000000000 === CONT TestMultitenantCompactor_ShouldStopCompactingTenantOnReachingMaxCompactionTime === CONT TestMultitenantCompactor_ShouldRetryCompactionOnFailureWhileDiscoveringUsersFromBucket === RUN TestSyncer_GarbageCollect_e2e/filesystem === PAUSE TestSyncer_GarbageCollect_e2e/filesystem === CONT TestMultitenantCompactor_ShouldIterateOverUsersAndRunCompaction --- PASS: TestMultitenantCompactor_ShouldRetryCompactionOnFailureWhileDiscoveringUsersFromBucket (0.05s) === CONT TestMultitenantCompactor_ShouldIncrementCompactionErrorIfFailedToCompactASingleTenant --- PASS: TestMultitenantCompactor_ShouldIterateOverUsersAndRunCompaction (0.29s) === CONT TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly === RUN TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly/inmem create 00000000000000000000000000 create 00000000010000000000000000 create 00000000340000000000000000 level=debug msg="deleted file" file=00000000000000000000000000/meta.json bucket=inmem level=debug msg="deleted file" file=00000000000000000000000000/deletion-mark.json bucket=inmem level=debug msg="deleted file" file=00000000010000000000000000/meta.json bucket=inmem level=debug msg="deleted file" file=00000000010000000000000000/deletion-mark.json bucket=inmem === RUN TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly/filesystem === PAUSE TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly/filesystem === CONT TestMultitenantCompactor_ShouldDoNothingOnNoUserBlocks --- PASS: TestMultitenantCompactor_ShouldDoNothingOnNoUserBlocks (0.04s) === CONT TestMultitenantCompactor_ShouldCompactAllUsersOnShardingEnabledButOnlyOneInstanceRunning --- PASS: TestMultitenantCompactor_ShouldIncrementCompactionErrorIfFailedToCompactASingleTenant (0.63s) === CONT TestMultitenantCompactor_ShouldSkipCompactionForJobsNoMoreOwnedAfterPlanning --- PASS: TestMultitenantCompactor_ShouldCompactAllUsersOnShardingEnabledButOnlyOneInstanceRunning (0.34s) === CONT TestMultitenantCompactor_ShouldCompactOnlyUsersOwnedByTheInstanceOnShardingEnabledAndMultipleInstancesRunning --- PASS: TestMultitenantCompactor_ShouldSkipCompactionForJobsNoMoreOwnedAfterPlanning (0.13s) === CONT TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForNoCompaction --- PASS: TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForNoCompaction (0.24s) === CONT TestMultitenantCompactor_ShouldNotCompactBlocksForUsersMarkedForDeletion --- PASS: TestMultitenantCompactor_ShouldNotCompactBlocksForUsersMarkedForDeletion (0.20s) === CONT TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForDeletion --- PASS: TestMultitenantCompactor_ShouldStopCompactingTenantOnReachingMaxCompactionTime (1.25s) === CONT TestGroupCompactE2E === RUN TestGroupCompactE2E/inmem level=info msg="start sync of metas" level=info msg="start of GC" level=info msg="start of compactions" level=info msg="compaction iterations done" --- PASS: TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForDeletion (0.28s) === CONT TestSyncer_GarbageCollect_e2e/filesystem create 00000000000000000000000000 create 00000000010000000000000000 create 00000000020000000000000000 create 00000000030000000000000000 create 00000000040000000000000000 create 00000000050000000000000000 create 00000000060000000000000000 create 00000000070000000000000000 create 00000000080000000000000000 create 00000000090000000000000000 create 00000000340000000000000000 create 00000000680000000000000000 create 000000009C0000000000000000 create 00000000CG0000000000000000 create 00000000FM0000000000000000 --- PASS: TestSyncer_GarbageCollect_e2e (0.03s) --- PASS: TestSyncer_GarbageCollect_e2e/inmem (0.03s) --- PASS: TestSyncer_GarbageCollect_e2e/filesystem (0.03s) === CONT TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly/filesystem create 00000000000000000000000000 create 00000000010000000000000000 create 00000000340000000000000000 level=debug msg="deleted file" file=00000000000000000000000000/meta.json bucket="fs: /tmp/TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnlyfilesystem4254626932/001" level=debug msg="deleted file" file=00000000000000000000000000/deletion-mark.json bucket="fs: /tmp/TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnlyfilesystem4254626932/001" level=debug msg="deleted file" file=00000000010000000000000000/meta.json bucket="fs: /tmp/TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnlyfilesystem4254626932/001" level=debug msg="deleted file" file=00000000010000000000000000/deletion-mark.json bucket="fs: /tmp/TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnlyfilesystem4254626932/001" --- PASS: TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly (0.01s) --- PASS: TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly/inmem (0.01s) --- PASS: TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly/filesystem (0.01s) --- PASS: TestMultitenantCompactor_ShouldFailCompactionOnTimeout (5.01s) === CONT TestGroupCompactE2E/inmem bucket_compactor_e2e_test.go:476: Error Trace: /build/mimir/src/mimir/pkg/compactor/bucket_compactor_e2e_test.go:476 /build/mimir/src/mimir/pkg/compactor/bucket_compactor_e2e_test.go:452 /build/mimir/src/mimir/pkg/compactor/bucket_compactor_e2e_test.go:270 /build/mimir/src/mimir/pkg/compactor/bucket_compactor_e2e_test.go:592 Error: Received unexpected error: write block: 2 errors: populate block: writing block: add series: context deadline exceeded; context deadline exceeded Test: TestGroupCompactE2E/inmem --- FAIL: TestGroupCompactE2E (24.38s) --- FAIL: TestGroupCompactE2E/inmem (24.38s) --- PASS: TestMultitenantCompactor_ShouldCompactOnlyUsersOwnedByTheInstanceOnShardingEnabledAndMultipleInstancesRunning (26.75s) FAIL FAIL github.com/grafana/mimir/pkg/compactor 560.623s === RUN TestClient_WriteSeries === RUN TestClient_WriteSeries/write_series_in_a_single_batch === RUN TestClient_WriteSeries/write_series_in_multiple_batches === RUN TestClient_WriteSeries/request_failed_with_4xx_error === RUN TestClient_WriteSeries/request_failed_with_5xx_error --- PASS: TestClient_WriteSeries (0.02s) --- PASS: TestClient_WriteSeries/write_series_in_a_single_batch (0.01s) --- PASS: TestClient_WriteSeries/write_series_in_multiple_batches (0.00s) --- PASS: TestClient_WriteSeries/request_failed_with_4xx_error (0.00s) --- PASS: TestClient_WriteSeries/request_failed_with_5xx_error (0.00s) === RUN TestClient_QueryRange === RUN TestClient_QueryRange/results_cache_not_explicitly_disabled === RUN TestClient_QueryRange/results_cache_disabled --- PASS: TestClient_QueryRange (0.01s) --- PASS: TestClient_QueryRange/results_cache_not_explicitly_disabled (0.00s) --- PASS: TestClient_QueryRange/results_cache_disabled (0.00s) === RUN TestClient_Query === RUN TestClient_Query/results_cache_not_explicitly_disabled === RUN TestClient_Query/results_cache_disabled --- PASS: TestClient_Query (0.00s) --- PASS: TestClient_Query/results_cache_not_explicitly_disabled (0.00s) --- PASS: TestClient_Query/results_cache_disabled (0.00s) === RUN TestManager_PeriodicRun --- PASS: TestManager_PeriodicRun (0.05s) === RUN TestManager_SmokeTest === RUN TestManager_SmokeTest/successful_smoke_test === RUN TestManager_SmokeTest/failed_smoke_test --- PASS: TestManager_SmokeTest (0.00s) --- PASS: TestManager_SmokeTest/successful_smoke_test (0.00s) --- PASS: TestManager_SmokeTest/failed_smoke_test (0.00s) === RUN TestAlignTimestampToInterval --- PASS: TestAlignTimestampToInterval (0.00s) === RUN TestGetQueryStep === RUN TestGetQueryStep/should_return_write_interval_if_expected_number_of_samples_is_<_1000 === RUN TestGetQueryStep/should_align_step_to_write_interval_and_guarantee_no_more_than_1000_samples --- PASS: TestGetQueryStep (0.00s) --- PASS: TestGetQueryStep/should_return_write_interval_if_expected_number_of_samples_is_<_1000 (0.00s) --- PASS: TestGetQueryStep/should_align_step_to_write_interval_and_guarantee_no_more_than_1000_samples (0.00s) === RUN TestVerifySineWaveSamplesSum === RUN TestVerifySineWaveSamplesSum/should_return_error_if_the_2nd_last_sample_has_an_unexpected_timestamp === RUN TestVerifySineWaveSamplesSum/should_return_no_error_if_all_samples_value_and_timestamp_match_the_expected_one_(1_series) === RUN TestVerifySineWaveSamplesSum/should_return_no_error_if_all_samples_value_and_timestamp_match_the_expected_one_(multiple_series) === RUN TestVerifySineWaveSamplesSum/should_return_error_if_there's_a_missing_series === RUN TestVerifySineWaveSamplesSum/should_return_error_if_there's_a_missing_sample --- PASS: TestVerifySineWaveSamplesSum (0.00s) --- PASS: TestVerifySineWaveSamplesSum/should_return_error_if_the_2nd_last_sample_has_an_unexpected_timestamp (0.00s) --- PASS: TestVerifySineWaveSamplesSum/should_return_no_error_if_all_samples_value_and_timestamp_match_the_expected_one_(1_series) (0.00s) --- PASS: TestVerifySineWaveSamplesSum/should_return_no_error_if_all_samples_value_and_timestamp_match_the_expected_one_(multiple_series) (0.00s) --- PASS: TestVerifySineWaveSamplesSum/should_return_error_if_there's_a_missing_series (0.00s) --- PASS: TestVerifySineWaveSamplesSum/should_return_error_if_there's_a_missing_sample (0.00s) === RUN TestMinTime --- PASS: TestMinTime (0.00s) === RUN TestMaxTime --- PASS: TestMaxTime (0.00s) === RUN TestRandTime --- PASS: TestRandTime (0.00s) === RUN TestWriteReadSeriesTest_Run === RUN TestWriteReadSeriesTest_Run/should_write_series_with_current_timestamp_if_it's_already_aligned_to_write_interval === RUN TestWriteReadSeriesTest_Run/should_write_series_with_timestamp_aligned_to_write_interval === RUN TestWriteReadSeriesTest_Run/should_write_series_from_last_written_timestamp_until_now === RUN TestWriteReadSeriesTest_Run/should_stop_remote_writing_on_network_error === RUN TestWriteReadSeriesTest_Run/should_stop_remote_writing_on_5xx_error === RUN TestWriteReadSeriesTest_Run/should_keep_remote_writing_next_intervals_on_4xx_error === RUN TestWriteReadSeriesTest_Run/should_query_written_series,_compare_results_and_track_no_failure_if_results_match === RUN TestWriteReadSeriesTest_Run/should_query_written_series,_compare_results_and_track_failure_if_results_don't_match --- PASS: TestWriteReadSeriesTest_Run (4.08s) --- PASS: TestWriteReadSeriesTest_Run/should_write_series_with_current_timestamp_if_it's_already_aligned_to_write_interval (0.01s) --- PASS: TestWriteReadSeriesTest_Run/should_write_series_with_timestamp_aligned_to_write_interval (0.01s) --- PASS: TestWriteReadSeriesTest_Run/should_write_series_from_last_written_timestamp_until_now (2.02s) --- PASS: TestWriteReadSeriesTest_Run/should_stop_remote_writing_on_network_error (0.00s) --- PASS: TestWriteReadSeriesTest_Run/should_stop_remote_writing_on_5xx_error (0.00s) --- PASS: TestWriteReadSeriesTest_Run/should_keep_remote_writing_next_intervals_on_4xx_error (2.01s) --- PASS: TestWriteReadSeriesTest_Run/should_query_written_series,_compare_results_and_track_no_failure_if_results_match (0.01s) --- PASS: TestWriteReadSeriesTest_Run/should_query_written_series,_compare_results_and_track_failure_if_results_don't_match (0.02s) === RUN TestWriteReadSeriesTest_Init === RUN TestWriteReadSeriesTest_Init/no_previously_written_samples_found === RUN TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-2h,_-1m] === RUN TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-36h,_-1m] === RUN TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-36h,_-1m]_but_last_data_point_of_previous_24h_period_is_missing === RUN TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-24h,_-1m] === RUN TestWriteReadSeriesTest_Init/the_configured_query_max_age_is_>_24h === RUN TestWriteReadSeriesTest_Init/the_configured_query_max_age_is_<_24h === RUN TestWriteReadSeriesTest_Init/the_most_recent_previously_written_data_point_is_older_than_1h_ago === RUN TestWriteReadSeriesTest_Init/the_first_query_fails === RUN TestWriteReadSeriesTest_Init/a_subsequent_query_fails === RUN TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_in_the_middle_of_the_last_24h_period === RUN TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_in_the_middle_of_the_previous_24h_period === RUN TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_exactly_at_the_beginning_of_this_24h_period --- PASS: TestWriteReadSeriesTest_Init (0.25s) --- PASS: TestWriteReadSeriesTest_Init/no_previously_written_samples_found (0.00s) --- PASS: TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-2h,_-1m] (0.00s) --- PASS: TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-36h,_-1m] (0.02s) --- PASS: TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-36h,_-1m]_but_last_data_point_of_previous_24h_period_is_missing (0.02s) --- PASS: TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-24h,_-1m] (0.02s) --- PASS: TestWriteReadSeriesTest_Init/the_configured_query_max_age_is_>_24h (0.09s) --- PASS: TestWriteReadSeriesTest_Init/the_configured_query_max_age_is_<_24h (0.00s) --- PASS: TestWriteReadSeriesTest_Init/the_most_recent_previously_written_data_point_is_older_than_1h_ago (0.00s) --- PASS: TestWriteReadSeriesTest_Init/the_first_query_fails (0.00s) --- PASS: TestWriteReadSeriesTest_Init/a_subsequent_query_fails (0.01s) --- PASS: TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_in_the_middle_of_the_last_24h_period (0.00s) --- PASS: TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_in_the_middle_of_the_previous_24h_period (0.02s) --- PASS: TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_exactly_at_the_beginning_of_this_24h_period (0.03s) === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min/max_query_time_has_not_been_set_yet === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min/max_query_time_is_older_than_max_age === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_query_time_=_max_query_time === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_and_max_query_time_are_within_the_last_1h === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_and_max_query_time_are_within_the_last_2h === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_query_time_is_older_than_24h === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/max_query_time_is_older_than_24h_but_more_recent_than_max_query_age === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_query_time_is_older_than_24h_but_max_query_age_is_only_10m --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min/max_query_time_has_not_been_set_yet (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min/max_query_time_is_older_than_max_age (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_query_time_=_max_query_time (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_and_max_query_time_are_within_the_last_1h (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_and_max_query_time_are_within_the_last_2h (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_query_time_is_older_than_24h (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/max_query_time_is_older_than_24h_but_more_recent_than_max_query_age (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_query_time_is_older_than_24h_but_max_query_age_is_only_10m (0.00s) PASS ok github.com/grafana/mimir/pkg/continuoustest 4.579s === RUN TestConfig_Validate === RUN TestConfig_Validate/should_fail_if_the_default_shard_size_is_negative === RUN TestConfig_Validate/should_pass_if_the_default_shard_size_>=_0 === RUN TestConfig_Validate/default_config_should_pass --- PASS: TestConfig_Validate (0.02s) --- PASS: TestConfig_Validate/should_fail_if_the_default_shard_size_is_negative (0.00s) --- PASS: TestConfig_Validate/should_pass_if_the_default_shard_size_>=_0 (0.01s) --- PASS: TestConfig_Validate/default_config_should_pass (0.00s) === RUN TestDistributor_Push === RUN TestDistributor_Push/A_push_of_no_samples_shouldn't_block_or_return_error,_even_if_ingesters_are_sad === RUN TestDistributor_Push/A_push_to_ingesters_with_an_old_sample_should_report_the_correct_metrics_with_no_metadata === RUN TestDistributor_Push/A_push_to_ingesters_with_a_current_sample_should_report_the_correct_metrics_with_no_metadata === RUN TestDistributor_Push/A_push_to_3_happy_ingesters_should_succeed === RUN TestDistributor_Push/A_push_to_2_happy_ingesters_should_succeed === RUN TestDistributor_Push/A_push_to_1_happy_ingesters_should_fail === RUN TestDistributor_Push/A_push_to_0_happy_ingesters_should_fail === RUN TestDistributor_Push/A_push_exceeding_burst_size_should_fail === RUN TestDistributor_Push/A_push_to_ingesters_without_samples_should_report_the_correct_metrics === RUN TestDistributor_Push/A_timed_out_push_should_fail --- PASS: TestDistributor_Push (1.57s) --- PASS: TestDistributor_Push/A_push_of_no_samples_shouldn't_block_or_return_error,_even_if_ingesters_are_sad (0.19s) --- PASS: TestDistributor_Push/A_push_to_ingesters_with_an_old_sample_should_report_the_correct_metrics_with_no_metadata (0.13s) --- PASS: TestDistributor_Push/A_push_to_ingesters_with_a_current_sample_should_report_the_correct_metrics_with_no_metadata (0.14s) --- PASS: TestDistributor_Push/A_push_to_3_happy_ingesters_should_succeed (0.14s) --- PASS: TestDistributor_Push/A_push_to_2_happy_ingesters_should_succeed (0.14s) --- PASS: TestDistributor_Push/A_push_to_1_happy_ingesters_should_fail (0.13s) --- PASS: TestDistributor_Push/A_push_to_0_happy_ingesters_should_fail (0.21s) --- PASS: TestDistributor_Push/A_push_exceeding_burst_size_should_fail (0.18s) --- PASS: TestDistributor_Push/A_push_to_ingesters_without_samples_should_report_the_correct_metrics (0.19s) --- PASS: TestDistributor_Push/A_timed_out_push_should_fail (0.12s) === RUN TestDistributor_ContextCanceledRequest --- PASS: TestDistributor_ContextCanceledRequest (0.12s) === RUN TestDistributor_MetricsCleanup --- PASS: TestDistributor_MetricsCleanup (0.14s) === RUN TestDistributor_PushRequestRateLimiter === RUN TestDistributor_PushRequestRateLimiter/request_limit_should_be_evenly_shared_across_distributors === RUN TestDistributor_PushRequestRateLimiter/request_limit_is_disabled_when_set_to_0 === RUN TestDistributor_PushRequestRateLimiter/request_burst_should_set_to_each_distributor --- PASS: TestDistributor_PushRequestRateLimiter (0.60s) --- PASS: TestDistributor_PushRequestRateLimiter/request_limit_should_be_evenly_shared_across_distributors (0.25s) --- PASS: TestDistributor_PushRequestRateLimiter/request_limit_is_disabled_when_set_to_0 (0.22s) --- PASS: TestDistributor_PushRequestRateLimiter/request_burst_should_set_to_each_distributor (0.13s) === RUN TestDistributor_PushIngestionRateLimiter === RUN TestDistributor_PushIngestionRateLimiter/evenly_share_the_ingestion_limit_across_distributors === RUN TestDistributor_PushIngestionRateLimiter/for_each_distributor,_set_an_ingestion_burst_limit. --- PASS: TestDistributor_PushIngestionRateLimiter (0.46s) --- PASS: TestDistributor_PushIngestionRateLimiter/evenly_share_the_ingestion_limit_across_distributors (0.22s) --- PASS: TestDistributor_PushIngestionRateLimiter/for_each_distributor,_set_an_ingestion_burst_limit. (0.24s) === RUN TestDistributor_PushInstanceLimits === RUN TestDistributor_PushInstanceLimits/hits_inflight_limit === RUN TestDistributor_PushInstanceLimits/below_ingestion_rate_limit === RUN TestDistributor_PushInstanceLimits/hits_rate_limit_on_first_request,_but_second_request_can_proceed === RUN TestDistributor_PushInstanceLimits/below_rate_limit_on_first_request,_but_hits_the_rate_limit_afterwards === RUN TestDistributor_PushInstanceLimits/below_inflight_size_limit === RUN TestDistributor_PushInstanceLimits/hits_inflight_size_limit === RUN TestDistributor_PushInstanceLimits/no_limits_limit === RUN TestDistributor_PushInstanceLimits/below_inflight_limit --- PASS: TestDistributor_PushInstanceLimits (2.14s) --- PASS: TestDistributor_PushInstanceLimits/hits_inflight_limit (0.13s) --- PASS: TestDistributor_PushInstanceLimits/below_ingestion_rate_limit (0.18s) --- PASS: TestDistributor_PushInstanceLimits/hits_rate_limit_on_first_request,_but_second_request_can_proceed (0.13s) --- PASS: TestDistributor_PushInstanceLimits/below_rate_limit_on_first_request,_but_hits_the_rate_limit_afterwards (0.99s) --- PASS: TestDistributor_PushInstanceLimits/below_inflight_size_limit (0.21s) --- PASS: TestDistributor_PushInstanceLimits/hits_inflight_size_limit (0.19s) --- PASS: TestDistributor_PushInstanceLimits/no_limits_limit (0.17s) --- PASS: TestDistributor_PushInstanceLimits/below_inflight_limit (0.13s) === RUN TestDistributor_PushHAInstances === RUN TestDistributor_PushHAInstances/0 === RUN TestDistributor_PushHAInstances/1 === RUN TestDistributor_PushHAInstances/2 === RUN TestDistributor_PushHAInstances/3 --- PASS: TestDistributor_PushHAInstances (6.14s) --- PASS: TestDistributor_PushHAInstances/0 (2.01s) --- PASS: TestDistributor_PushHAInstances/1 (2.01s) --- PASS: TestDistributor_PushHAInstances/2 (0.12s) --- PASS: TestDistributor_PushHAInstances/3 (2.01s) === RUN TestDistributor_PushQuery === RUN TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=5) --- PASS: TestDistributor_PushQuery (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=4) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=7) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=6) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=9) (0.23s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=4) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) (0.20s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=5)) (0.15s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=0)) (0.17s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=5)) (0.22s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=0)) (0.15s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=0)) (0.18s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=5)) (0.16s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=9) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=7) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=8) (0.18s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=6) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=4) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=2) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=7) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=9) (0.22s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=5) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=4) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=1) (0.19s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=8) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=6) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=7) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=5) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=2) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=1) (0.21s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=8) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=9) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=7) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=6) (0.18s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=5) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=4) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=3) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=9) (0.19s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=7) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=6) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=4) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=3) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=2) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=0) (0.11s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=1) (0.22s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=7) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=8) (0.19s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=6) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=7) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=3) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=4) (0.19s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=2) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=8) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=9) (0.18s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) (0.22s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=2) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=5) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=6) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=3) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=4) (0.18s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=0) (0.14s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=9) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=8) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=7) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=0)) (0.20s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=5) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=6) (0.22s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=4) (0.20s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=1) (0.15s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=2) (0.21s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=9) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=7) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=5) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=4) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=3) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=1) (0.14s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) (0.17s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=9) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=7) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=6) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=4) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=5) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=3) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=2) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=0) (0.14s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=8) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=7) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=6) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=5) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=4) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=1) (0.15s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=2) (0.21s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=0) (0.16s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=5)) (0.19s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=5)) (0.16s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=8) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=5) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=7) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=4) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=3) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=9) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=7) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=4) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=5) (0.22s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=3) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=2) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=0) (0.25s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=9) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=8) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=7) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=6) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=4) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=5) (0.20s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=3) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=2) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=0) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=1) (0.23s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) (0.20s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=7) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=9) (0.21s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=4) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=3) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=1) (0.14s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) (0.15s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=0)) (0.15s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=8) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=7) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=4) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=3) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=1) (0.19s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=9) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=7) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=6) (0.18s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=3) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=4) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=2) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=1) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=0) (0.14s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) (0.18s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=9) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=8) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=6) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=7) (0.19s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=4) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=2) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=5) (0.23s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=3) (0.21s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=1) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) (0.18s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=9) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=8) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=6) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=7) (0.19s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=5) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=4) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=2) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=0) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=5)) (0.16s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=9) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=0)) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=8) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=7) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=4) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=1) (0.15s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=9) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=7) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=4) (0.19s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=3) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=2) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=5)) (0.16s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=0)) (0.16s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=0)) (0.18s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=5)) (0.22s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=5)) (0.16s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=0)) (0.17s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=9) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=8) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=6) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=7) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=5) (0.21s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=4) (0.20s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=3) (0.20s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=0) (0.18s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=9) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=8) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=7) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=5) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=6) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=4) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=3) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=2) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) (0.15s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=9) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=8) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=7) (0.21s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=5) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=4) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=2) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=1) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=0) (0.21s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) (0.17s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) (0.16s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=9) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=7) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=3) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=4) (0.19s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=5) (0.26s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=1) (0.23s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=0) (0.18s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=7) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=6) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=4) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=2) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=9) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=7) (0.18s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=4) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=3) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=9) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=7) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=4) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=3) (0.19s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=9) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=7) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=6) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=4) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=3) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=2) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) (0.15s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) (0.16s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=0)) (0.16s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=0)) (0.15s) === RUN TestDistributor_QueryStream_ShouldReturnErrorIfMaxChunksPerQueryLimitIsReached --- PASS: TestDistributor_QueryStream_ShouldReturnErrorIfMaxChunksPerQueryLimitIsReached (0.14s) === RUN TestDistributor_QueryStream_ShouldReturnErrorIfMaxSeriesPerQueryLimitIsReached --- PASS: TestDistributor_QueryStream_ShouldReturnErrorIfMaxSeriesPerQueryLimitIsReached (0.20s) === RUN TestDistributor_QueryStream_ShouldReturnErrorIfMaxChunkBytesPerQueryLimitIsReached --- PASS: TestDistributor_QueryStream_ShouldReturnErrorIfMaxChunkBytesPerQueryLimitIsReached (0.15s) === RUN TestDistributor_Push_LabelRemoval --- PASS: TestDistributor_Push_LabelRemoval (0.87s) === RUN TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime === RUN TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_1_and_dropped_HA_replica_label === RUN TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_2_with_value_1 === RUN TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_2 === RUN TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_1 === RUN TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_1_and_dropped_label_due_to_config --- PASS: TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime (0.79s) --- PASS: TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_1_and_dropped_HA_replica_label (0.16s) --- PASS: TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_2_with_value_1 (0.18s) --- PASS: TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_2 (0.13s) --- PASS: TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_1 (0.19s) --- PASS: TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_1_and_dropped_label_due_to_config (0.13s) === RUN TestDistributor_Push_LabelNameValidation === RUN TestDistributor_Push_LabelNameValidation/label_name_validation_can_be_skipped_via_config === RUN TestDistributor_Push_LabelNameValidation/label_name_validation_can_be_skipped_via_WriteRequest_parameter === RUN TestDistributor_Push_LabelNameValidation/label_name_validation_is_on_by_default --- PASS: TestDistributor_Push_LabelNameValidation (0.38s) --- PASS: TestDistributor_Push_LabelNameValidation/label_name_validation_can_be_skipped_via_config (0.15s) --- PASS: TestDistributor_Push_LabelNameValidation/label_name_validation_can_be_skipped_via_WriteRequest_parameter (0.11s) --- PASS: TestDistributor_Push_LabelNameValidation/label_name_validation_is_on_by_default (0.12s) === RUN TestDistributor_Push_ExemplarValidation === RUN TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_too_many_series_labels === RUN TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_duplicate_series_labels === RUN TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_empty_series_label_name === RUN TestDistributor_Push_ExemplarValidation/valid_exemplar === RUN TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_no_labels === RUN TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_no_timestamp === RUN TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_too_long_labelset --- PASS: TestDistributor_Push_ExemplarValidation (1.03s) --- PASS: TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_too_many_series_labels (0.13s) --- PASS: TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_duplicate_series_labels (0.16s) --- PASS: TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_empty_series_label_name (0.20s) --- PASS: TestDistributor_Push_ExemplarValidation/valid_exemplar (0.12s) --- PASS: TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_no_labels (0.17s) --- PASS: TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_no_timestamp (0.12s) --- PASS: TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_too_long_labelset (0.13s) === RUN TestDistributor_ExemplarValidation === RUN TestDistributor_ExemplarValidation/valid_exemplars === RUN TestDistributor_ExemplarValidation/one_old,_one_new,_separate_series === RUN TestDistributor_ExemplarValidation/multi_exemplars === RUN TestDistributor_ExemplarValidation/one_old,_one_new,_same_series === RUN TestDistributor_ExemplarValidation/disable_exemplars --- PASS: TestDistributor_ExemplarValidation (0.72s) --- PASS: TestDistributor_ExemplarValidation/valid_exemplars (0.11s) --- PASS: TestDistributor_ExemplarValidation/one_old,_one_new,_separate_series (0.21s) --- PASS: TestDistributor_ExemplarValidation/multi_exemplars (0.16s) --- PASS: TestDistributor_ExemplarValidation/one_old,_one_new,_same_series (0.12s) --- PASS: TestDistributor_ExemplarValidation/disable_exemplars (0.11s) === RUN TestSlowQueries === RUN TestSlowQueries/0 === RUN TestSlowQueries/1 === RUN TestSlowQueries/2 === RUN TestSlowQueries/3 --- PASS: TestSlowQueries (0.95s) --- PASS: TestSlowQueries/0 (0.22s) --- PASS: TestSlowQueries/1 (0.22s) --- PASS: TestSlowQueries/2 (0.28s) --- PASS: TestSlowQueries/3 (0.23s) === RUN TestDistributor_MetricsForLabelMatchers === RUN TestDistributor_MetricsForLabelMatchers/should_return_an_empty_response_if_no_metric_match === RUN TestDistributor_MetricsForLabelMatchers/should_filter_metrics_by_single_matcher === RUN TestDistributor_MetricsForLabelMatchers/should_filter_metrics_by_multiple_matchers === RUN TestDistributor_MetricsForLabelMatchers/should_return_all_matching_metrics_even_if_their_FastFingerprint_collide === RUN TestDistributor_MetricsForLabelMatchers/should_query_only_ingesters_belonging_to_tenant's_subring_if_shuffle_shard_size_is_set --- PASS: TestDistributor_MetricsForLabelMatchers (0.76s) --- PASS: TestDistributor_MetricsForLabelMatchers/should_return_an_empty_response_if_no_metric_match (0.14s) --- PASS: TestDistributor_MetricsForLabelMatchers/should_filter_metrics_by_single_matcher (0.21s) --- PASS: TestDistributor_MetricsForLabelMatchers/should_filter_metrics_by_multiple_matchers (0.12s) --- PASS: TestDistributor_MetricsForLabelMatchers/should_return_all_matching_metrics_even_if_their_FastFingerprint_collide (0.12s) --- PASS: TestDistributor_MetricsForLabelMatchers/should_query_only_ingesters_belonging_to_tenant's_subring_if_shuffle_shard_size_is_set (0.18s) === RUN TestDistributor_LabelNames === RUN TestDistributor_LabelNames/should_return_an_empty_response_if_no_metric_match === RUN TestDistributor_LabelNames/should_filter_metrics_by_single_matcher === RUN TestDistributor_LabelNames/should_filter_metrics_by_multiple_matchers === RUN TestDistributor_LabelNames/should_query_only_ingesters_belonging_to_tenant's_subring_if_shuffle_sharding_is_enabled --- PASS: TestDistributor_LabelNames (0.55s) --- PASS: TestDistributor_LabelNames/should_return_an_empty_response_if_no_metric_match (0.12s) --- PASS: TestDistributor_LabelNames/should_filter_metrics_by_single_matcher (0.18s) --- PASS: TestDistributor_LabelNames/should_filter_metrics_by_multiple_matchers (0.12s) --- PASS: TestDistributor_LabelNames/should_query_only_ingesters_belonging_to_tenant's_subring_if_shuffle_sharding_is_enabled (0.12s) === RUN TestDistributor_MetricsMetadata === RUN TestDistributor_MetricsMetadata/should_query_only_ingesters_belonging_to_tenant's_subring_if_shuffle_sharding_is_enabled === RUN TestDistributor_MetricsMetadata/should_query_all_ingesters_if_shuffle_sharding_is_enabled_but_shard_size_is_0 --- PASS: TestDistributor_MetricsMetadata (0.24s) --- PASS: TestDistributor_MetricsMetadata/should_query_only_ingesters_belonging_to_tenant's_subring_if_shuffle_sharding_is_enabled (0.12s) --- PASS: TestDistributor_MetricsMetadata/should_query_all_ingesters_if_shuffle_sharding_is_enabled_but_shard_size_is_0 (0.12s) === RUN TestDistributor_LabelNamesAndValuesLimitTest === RUN TestDistributor_LabelNamesAndValuesLimitTest/expected_no_error_if_sizeLimit_is_not_reached === RUN TestDistributor_LabelNamesAndValuesLimitTest/expected_error_if_sizeLimit_is_reached --- PASS: TestDistributor_LabelNamesAndValuesLimitTest (0.23s) --- PASS: TestDistributor_LabelNamesAndValuesLimitTest/expected_no_error_if_sizeLimit_is_not_reached (0.12s) --- PASS: TestDistributor_LabelNamesAndValuesLimitTest/expected_error_if_sizeLimit_is_reached (0.12s) === RUN TestDistributor_LabelNamesAndValues === RUN TestDistributor_LabelNamesAndValues/should_group_values_of_labels_by_label_name_and_return_only_distinct_label_values === RUN TestDistributor_LabelNamesAndValues/should_return_the_results_if_zone_awareness_is_enabled_and_only_2_zones_return_the_results --- PASS: TestDistributor_LabelNamesAndValues (0.24s) --- PASS: TestDistributor_LabelNamesAndValues/should_group_values_of_labels_by_label_name_and_return_only_distinct_label_values (0.12s) --- PASS: TestDistributor_LabelNamesAndValues/should_return_the_results_if_zone_awareness_is_enabled_and_only_2_zones_return_the_results (0.12s) === RUN TestDistributor_LabelValuesCardinality_ExpectedAllIngestersResponsesToBeCompleted --- PASS: TestDistributor_LabelValuesCardinality_ExpectedAllIngestersResponsesToBeCompleted (8.99s) === RUN TestDistributor_LabelNamesAndValues_ExpectedAllPossibleLabelNamesAndValuesToBeReturned --- PASS: TestDistributor_LabelNamesAndValues_ExpectedAllPossibleLabelNamesAndValuesToBeReturned (11.04s) === RUN TestDistributor_IngestionIsControlledByForwarder === RUN TestDistributor_IngestionIsControlledByForwarder/do_ingest_with_only_samples === RUN TestDistributor_IngestionIsControlledByForwarder/don't_ingest_with_only_samples === RUN TestDistributor_IngestionIsControlledByForwarder/do_ingest_with_metadata === RUN TestDistributor_IngestionIsControlledByForwarder/don't_ingest_with_metadata === RUN TestDistributor_IngestionIsControlledByForwarder/do_ingest_with_exemplars === RUN TestDistributor_IngestionIsControlledByForwarder/don't_ingest_with_exemplars --- PASS: TestDistributor_IngestionIsControlledByForwarder (0.88s) --- PASS: TestDistributor_IngestionIsControlledByForwarder/do_ingest_with_only_samples (0.15s) --- PASS: TestDistributor_IngestionIsControlledByForwarder/don't_ingest_with_only_samples (0.12s) --- PASS: TestDistributor_IngestionIsControlledByForwarder/do_ingest_with_metadata (0.15s) --- PASS: TestDistributor_IngestionIsControlledByForwarder/don't_ingest_with_metadata (0.21s) --- PASS: TestDistributor_IngestionIsControlledByForwarder/do_ingest_with_exemplars (0.12s) --- PASS: TestDistributor_IngestionIsControlledByForwarder/don't_ingest_with_exemplars (0.13s) === RUN TestDistributor_LabelValuesCardinality === RUN TestDistributor_LabelValuesCardinality/should_return_a_map_with_the_label_values_and_series_occurrences_of_a_single_label_name === RUN TestDistributor_LabelValuesCardinality/should_return_a_map_with_the_label_values_and_series_occurrences_of_all_the_label_names === RUN TestDistributor_LabelValuesCardinality/should_return_an_empty_map_if_no_label_names --- PASS: TestDistributor_LabelValuesCardinality (0.37s) --- PASS: TestDistributor_LabelValuesCardinality/should_return_a_map_with_the_label_values_and_series_occurrences_of_a_single_label_name (0.12s) --- PASS: TestDistributor_LabelValuesCardinality/should_return_a_map_with_the_label_values_and_series_occurrences_of_all_the_label_names (0.13s) --- PASS: TestDistributor_LabelValuesCardinality/should_return_an_empty_map_if_no_label_names (0.13s) === RUN TestDistributor_LabelValuesCardinalityLimit === RUN TestDistributor_LabelValuesCardinalityLimit/should_return_a_httpgrpc_error_if_the_maximum_number_of_label_names_per_request_is_reached === RUN TestDistributor_LabelValuesCardinalityLimit/should_succeed_if_the_maximum_number_of_label_names_per_request_is_not_reached --- PASS: TestDistributor_LabelValuesCardinalityLimit (0.23s) --- PASS: TestDistributor_LabelValuesCardinalityLimit/should_return_a_httpgrpc_error_if_the_maximum_number_of_label_names_per_request_is_reached (0.12s) --- PASS: TestDistributor_LabelValuesCardinalityLimit/should_succeed_if_the_maximum_number_of_label_names_per_request_is_not_reached (0.12s) === RUN TestDistributor_LabelValuesCardinality_Concurrency === RUN TestDistributor_LabelValuesCardinality_Concurrency/should_fail_with_an_error_if_at_least_one_ingester's_LabelValuesCardinality_and/or_UserStats_operations_fails --- PASS: TestDistributor_LabelValuesCardinality_Concurrency (0.12s) --- PASS: TestDistributor_LabelValuesCardinality_Concurrency/should_fail_with_an_error_if_at_least_one_ingester's_LabelValuesCardinality_and/or_UserStats_operations_fails (0.12s) === RUN TestHaDedupeMiddleware === RUN TestHaDedupeMiddleware/no_changes_on_empty_request === RUN TestHaDedupeMiddleware/no_changes_if_accept_HA_samples_is_false === RUN TestHaDedupeMiddleware/remove_replica_label_with_HA_tracker_disabled === RUN TestHaDedupeMiddleware/do_nothing_without_user_in_context,_don't_even_call_next === RUN TestHaDedupeMiddleware/perform_HA_deduplication === RUN TestHaDedupeMiddleware/exceed_max_ha_clusters_limit --- PASS: TestHaDedupeMiddleware (6.43s) --- PASS: TestHaDedupeMiddleware/no_changes_on_empty_request (2.01s) --- PASS: TestHaDedupeMiddleware/no_changes_if_accept_HA_samples_is_false (2.01s) --- PASS: TestHaDedupeMiddleware/remove_replica_label_with_HA_tracker_disabled (0.15s) --- PASS: TestHaDedupeMiddleware/do_nothing_without_user_in_context,_don't_even_call_next (2.01s) --- PASS: TestHaDedupeMiddleware/perform_HA_deduplication (0.11s) --- PASS: TestHaDedupeMiddleware/exceed_max_ha_clusters_limit (0.15s) === RUN TestInstanceLimitsBeforeHaDedupe --- PASS: TestInstanceLimitsBeforeHaDedupe (2.01s) === RUN TestRelabelMiddleware === RUN TestRelabelMiddleware/do_nothing === RUN TestRelabelMiddleware/no_user_in_context === RUN TestRelabelMiddleware/apply_a_relabel_rule === RUN TestRelabelMiddleware/drop_two_out_of_three_labels === RUN TestRelabelMiddleware/drop_entire_series_if_they_have_no_labels --- PASS: TestRelabelMiddleware (0.68s) --- PASS: TestRelabelMiddleware/do_nothing (0.19s) --- PASS: TestRelabelMiddleware/no_user_in_context (0.15s) --- PASS: TestRelabelMiddleware/apply_a_relabel_rule (0.11s) --- PASS: TestRelabelMiddleware/drop_two_out_of_three_labels (0.12s) --- PASS: TestRelabelMiddleware/drop_entire_series_if_they_have_no_labels (0.11s) === RUN TestMarkEphemeralMiddleware === RUN TestMarkEphemeralMiddleware/half_-_half === RUN TestMarkEphemeralMiddleware/no_ephemeral === RUN TestMarkEphemeralMiddleware/all_ephemeral --- PASS: TestMarkEphemeralMiddleware (0.38s) --- PASS: TestMarkEphemeralMiddleware/half_-_half (0.11s) --- PASS: TestMarkEphemeralMiddleware/no_ephemeral (0.13s) --- PASS: TestMarkEphemeralMiddleware/all_ephemeral (0.13s) === RUN TestHaDedupeAndRelabelBeforeForwarding --- PASS: TestHaDedupeAndRelabelBeforeForwarding (0.13s) === RUN TestValidationBeforeForwarding --- PASS: TestValidationBeforeForwarding (2.01s) === RUN TestDistributorValidation === RUN TestDistributorValidation/0 === RUN TestDistributorValidation/1 === RUN TestDistributorValidation/2 === RUN TestDistributorValidation/3 === RUN TestDistributorValidation/4 === RUN TestDistributorValidation/5 === RUN TestDistributorValidation/6 --- PASS: TestDistributorValidation (0.97s) --- PASS: TestDistributorValidation/0 (0.12s) --- PASS: TestDistributorValidation/1 (0.12s) --- PASS: TestDistributorValidation/2 (0.20s) --- PASS: TestDistributorValidation/3 (0.13s) --- PASS: TestDistributorValidation/4 (0.14s) --- PASS: TestDistributorValidation/5 (0.14s) --- PASS: TestDistributorValidation/6 (0.12s) === RUN TestRemoveReplicaLabel --- PASS: TestRemoveReplicaLabel (0.00s) === RUN TestShardByAllLabelsReturnsWrongResultsForUnsortedLabels --- PASS: TestShardByAllLabelsReturnsWrongResultsForUnsortedLabels (0.00s) === RUN TestSortLabels --- PASS: TestSortLabels (0.00s) === RUN TestDistributor_Push_Relabel --- PASS: TestDistributor_Push_Relabel (0.23s) === RUN TestDistributor_MetricsWithRequestModifications === RUN TestDistributor_MetricsWithRequestModifications/No_modifications === RUN TestDistributor_MetricsWithRequestModifications/Drop_samples_via_relabel_rules === RUN TestDistributor_MetricsWithRequestModifications/Drop_samples_via_drop_label_rule === RUN TestDistributor_MetricsWithRequestModifications/Drop_requests_via_ha-dedupe === RUN TestDistributor_MetricsWithRequestModifications/Some_samples_get_forwarded === RUN TestDistributor_MetricsWithRequestModifications/Drop_examplars_via_limit === RUN TestDistributor_MetricsWithRequestModifications/Drop_half_of_samples_via_label_value_length_limit === RUN TestDistributor_MetricsWithRequestModifications/Drop_metadata_via_label_value_length_limit,_except_help --- PASS: TestDistributor_MetricsWithRequestModifications (2.43s) --- PASS: TestDistributor_MetricsWithRequestModifications/No_modifications (0.12s) --- PASS: TestDistributor_MetricsWithRequestModifications/Drop_samples_via_relabel_rules (0.17s) --- PASS: TestDistributor_MetricsWithRequestModifications/Drop_samples_via_drop_label_rule (0.13s) --- PASS: TestDistributor_MetricsWithRequestModifications/Drop_requests_via_ha-dedupe (0.18s) --- PASS: TestDistributor_MetricsWithRequestModifications/Some_samples_get_forwarded (0.12s) --- PASS: TestDistributor_MetricsWithRequestModifications/Drop_examplars_via_limit (0.12s) --- PASS: TestDistributor_MetricsWithRequestModifications/Drop_half_of_samples_via_label_value_length_limit (0.12s) --- PASS: TestDistributor_MetricsWithRequestModifications/Drop_metadata_via_label_value_length_limit,_except_help (0.12s) === RUN TestDistributor_CleanupIsDoneAfterLastIngesterReturns --- PASS: TestDistributor_CleanupIsDoneAfterLastIngesterReturns (0.14s) === RUN TestSeriesAreShardedToCorrectIngesters --- PASS: TestSeriesAreShardedToCorrectIngesters (0.35s) === RUN TestHATrackerConfig_Validate === PAUSE TestHATrackerConfig_Validate === RUN TestWatchPrefixAssignment --- PASS: TestWatchPrefixAssignment (0.00s) === RUN TestCheckReplicaOverwriteTimeout --- PASS: TestCheckReplicaOverwriteTimeout (2.00s) === RUN TestCheckReplicaMultiCluster --- PASS: TestCheckReplicaMultiCluster (2.00s) === RUN TestCheckReplicaMultiClusterTimeout --- PASS: TestCheckReplicaMultiClusterTimeout (2.00s) === RUN TestCheckReplicaUpdateTimeout --- PASS: TestCheckReplicaUpdateTimeout (2.00s) === RUN TestCheckReplicaMultiUser --- PASS: TestCheckReplicaMultiUser (2.00s) === RUN TestCheckReplicaUpdateTimeoutJitter === PAUSE TestCheckReplicaUpdateTimeoutJitter === RUN TestFindHALabels --- PASS: TestFindHALabels (0.00s) === RUN TestHATrackerConfig_ShouldCustomizePrefixDefaultValue --- PASS: TestHATrackerConfig_ShouldCustomizePrefixDefaultValue (0.00s) === RUN TestHAClustersLimit --- PASS: TestHAClustersLimit (0.06s) === RUN TestTooManyClustersError --- PASS: TestTooManyClustersError (0.00s) === RUN TestReplicasNotMatchError --- PASS: TestReplicasNotMatchError (0.00s) === RUN TestHATracker_MetricsCleanup --- PASS: TestHATracker_MetricsCleanup (0.00s) === RUN TestCheckReplicaCleanup --- PASS: TestCheckReplicaCleanup (2.00s) === RUN TestHealthyInstanceDelegate_OnRingInstanceHeartbeat === RUN TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/all_instances_healthy_and_active === RUN TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/all_instances_healthy_not_all_instances_active === RUN TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/some_instances_healthy_all_instances_active === RUN TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/some_instances_healthy_but_timeout_disabled_all_instances_active --- PASS: TestHealthyInstanceDelegate_OnRingInstanceHeartbeat (0.00s) --- PASS: TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/all_instances_healthy_and_active (0.00s) --- PASS: TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/all_instances_healthy_not_all_instances_active (0.00s) --- PASS: TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/some_instances_healthy_all_instances_active (0.00s) --- PASS: TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/some_instances_healthy_but_timeout_disabled_all_instances_active (0.00s) === RUN TestMergeSamplesIntoFirstDuplicates --- PASS: TestMergeSamplesIntoFirstDuplicates (0.00s) === RUN TestMergeSamplesIntoFirst --- PASS: TestMergeSamplesIntoFirst (0.00s) === RUN TestMergeSamplesIntoFirstNilA --- PASS: TestMergeSamplesIntoFirstNilA (0.00s) === RUN TestMergeSamplesIntoFirstNilB --- PASS: TestMergeSamplesIntoFirstNilB (0.00s) === RUN TestMergeExemplars === RUN TestMergeExemplars/test0 === RUN TestMergeExemplars/test1 === RUN TestMergeExemplars/test2 === RUN TestMergeExemplars/test3 === RUN TestMergeExemplars/test4 === RUN TestMergeExemplars/test5 === RUN TestMergeExemplars/test6 --- PASS: TestMergeExemplars (0.00s) --- PASS: TestMergeExemplars/test0 (0.00s) --- PASS: TestMergeExemplars/test1 (0.00s) --- PASS: TestMergeExemplars/test2 (0.00s) --- PASS: TestMergeExemplars/test3 (0.00s) --- PASS: TestMergeExemplars/test4 (0.00s) --- PASS: TestMergeExemplars/test5 (0.00s) --- PASS: TestMergeExemplars/test6 (0.00s) === RUN TestIngestionRateStrategy === RUN TestIngestionRateStrategy/rate_limiter_should_share_the_limit_across_the_number_of_distributors === RUN TestIngestionRateStrategy/infinite_rate_limiter_should_return_unlimited_settings --- PASS: TestIngestionRateStrategy (0.00s) --- PASS: TestIngestionRateStrategy/rate_limiter_should_share_the_limit_across_the_number_of_distributors (0.00s) --- PASS: TestIngestionRateStrategy/infinite_rate_limiter_should_return_unlimited_settings (0.00s) === CONT TestHATrackerConfig_Validate === CONT TestCheckReplicaUpdateTimeoutJitter === RUN TestCheckReplicaUpdateTimeoutJitter/should_not_refresh_the_replica_if_the_update_timeout_is_not_expired_yet_(without_jitter) === RUN TestCheckReplicaUpdateTimeoutJitter/should_refresh_the_replica_if_the_update_timeout_is_expired_(without_jitter) === RUN TestHATrackerConfig_Validate/should_pass_if_failover_timeout_is_>=_update_timeout_+_jitter_+_1_sec === RUN TestHATrackerConfig_Validate/should_fail_if_KV_backend_is_set_to_memberlist === RUN TestHATrackerConfig_Validate/should_pass_with_default_config === RUN TestHATrackerConfig_Validate/should_fail_if_max_update_timeout_jitter_is_negative === RUN TestHATrackerConfig_Validate/should_fail_if_failover_timeout_is_<_update_timeout_+_jitter_+_1_sec --- PASS: TestHATrackerConfig_Validate (0.01s) --- PASS: TestHATrackerConfig_Validate/should_pass_if_failover_timeout_is_>=_update_timeout_+_jitter_+_1_sec (0.00s) --- PASS: TestHATrackerConfig_Validate/should_fail_if_KV_backend_is_set_to_memberlist (0.00s) --- PASS: TestHATrackerConfig_Validate/should_pass_with_default_config (0.00s) --- PASS: TestHATrackerConfig_Validate/should_fail_if_max_update_timeout_jitter_is_negative (0.00s) --- PASS: TestHATrackerConfig_Validate/should_fail_if_failover_timeout_is_<_update_timeout_+_jitter_+_1_sec (0.00s) === RUN TestCheckReplicaUpdateTimeoutJitter/should_not_refresh_the_replica_if_the_update_timeout_is_not_expired_yet_(with_jitter) === RUN TestCheckReplicaUpdateTimeoutJitter/should_refresh_the_replica_if_the_update_timeout_is_expired_(with_jitter) --- PASS: TestCheckReplicaUpdateTimeoutJitter (3.01s) --- PASS: TestCheckReplicaUpdateTimeoutJitter/should_not_refresh_the_replica_if_the_update_timeout_is_not_expired_yet_(without_jitter) (0.00s) --- PASS: TestCheckReplicaUpdateTimeoutJitter/should_refresh_the_replica_if_the_update_timeout_is_expired_(without_jitter) (2.00s) --- PASS: TestCheckReplicaUpdateTimeoutJitter/should_not_refresh_the_replica_if_the_update_timeout_is_not_expired_yet_(with_jitter) (0.00s) --- PASS: TestCheckReplicaUpdateTimeoutJitter/should_refresh_the_replica_if_the_update_timeout_is_expired_(with_jitter) (1.00s) PASS ok github.com/grafana/mimir/pkg/distributor 87.204s ? github.com/grafana/mimir/pkg/distributor/distributorpb [no test files] === RUN TestForwardingSamplesSuccessfullyToSingleTarget --- PASS: TestForwardingSamplesSuccessfullyToSingleTarget (0.05s) === RUN TestForwardingOmitOldSamples === RUN TestForwardingOmitOldSamples/drop_nothing === RUN TestForwardingOmitOldSamples/drop_some_of_the_samples,_ingest_true === RUN TestForwardingOmitOldSamples/drop_some_of_the_samples,_ingest_false === RUN TestForwardingOmitOldSamples/split_one_sample_slice_in_the_middle === RUN TestForwardingOmitOldSamples/drop_all_of_the_samples --- PASS: TestForwardingOmitOldSamples (0.06s) --- PASS: TestForwardingOmitOldSamples/drop_nothing (0.01s) --- PASS: TestForwardingOmitOldSamples/drop_some_of_the_samples,_ingest_true (0.02s) --- PASS: TestForwardingOmitOldSamples/drop_some_of_the_samples,_ingest_false (0.01s) --- PASS: TestForwardingOmitOldSamples/split_one_sample_slice_in_the_middle (0.01s) --- PASS: TestForwardingOmitOldSamples/drop_all_of_the_samples (0.01s) === RUN TestForwardingEnsureThatPooledObjectsGetReturned === RUN TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_one_metric_and_don't_ingest_them === RUN TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_one_metric_and_ingest_them === RUN TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_and_don't_ingest_them === RUN TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_to_and_ingest_them === RUN TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_to_and_ingest_half_of_them_(1) === RUN TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_and_ingest_half_of_them_(2) === RUN TestForwardingEnsureThatPooledObjectsGetReturned/don't_forward_samples,_just_ingest_them --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned (0.07s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_one_metric_and_don't_ingest_them (0.01s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_one_metric_and_ingest_them (0.01s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_and_don't_ingest_them (0.01s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_to_and_ingest_them (0.01s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_to_and_ingest_half_of_them_(1) (0.01s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_and_ingest_half_of_them_(2) (0.02s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/don't_forward_samples,_just_ingest_them (0.00s) === RUN TestDropSamplesBefore === RUN TestDropSamplesBefore/no_samples === RUN TestDropSamplesBefore/all_samples_before_cutoff === RUN TestDropSamplesBefore/all_samples_after_cutoff === RUN TestDropSamplesBefore/cut_some_samples_off === RUN TestDropSamplesBefore/unsorted_sample_slice --- PASS: TestDropSamplesBefore (0.00s) --- PASS: TestDropSamplesBefore/no_samples (0.00s) --- PASS: TestDropSamplesBefore/all_samples_before_cutoff (0.00s) --- PASS: TestDropSamplesBefore/all_samples_after_cutoff (0.00s) --- PASS: TestDropSamplesBefore/cut_some_samples_off (0.00s) --- PASS: TestDropSamplesBefore/unsorted_sample_slice (0.00s) === RUN TestForwardingToHTTPGrpcTarget --- PASS: TestForwardingToHTTPGrpcTarget (0.03s) === RUN TestUsingPools --- PASS: TestUsingPools (0.00s) PASS ok github.com/grafana/mimir/pkg/distributor/forwarding 0.905s ? github.com/grafana/mimir/pkg/flusher [no test files] === RUN TestFrontend_RequestHostHeaderWhenDownstreamURLIsConfigured --- PASS: TestFrontend_RequestHostHeaderWhenDownstreamURLIsConfigured (0.05s) === RUN TestFrontend_LogsSlowQueriesFormValues --- PASS: TestFrontend_LogsSlowQueriesFormValues (0.04s) === RUN TestFrontend_ReturnsRequestBodyTooLargeError --- PASS: TestFrontend_ReturnsRequestBodyTooLargeError (0.02s) PASS ok github.com/grafana/mimir/pkg/frontend 0.631s === RUN TestRequest === RUN TestRequest/0 === RUN TestRequest/1 === RUN TestRequest/2 === RUN TestRequest/3 === RUN TestRequest/4 === RUN TestRequest/5 === RUN TestRequest/6 === RUN TestRequest/7 --- PASS: TestRequest (0.01s) --- PASS: TestRequest/0 (0.00s) --- PASS: TestRequest/1 (0.00s) --- PASS: TestRequest/2 (0.00s) --- PASS: TestRequest/3 (0.00s) --- PASS: TestRequest/4 (0.00s) --- PASS: TestRequest/5 (0.00s) --- PASS: TestRequest/6 (0.00s) --- PASS: TestRequest/7 (0.00s) === RUN TestDecodeFailedResponse === RUN TestDecodeFailedResponse/internal_error === RUN TestDecodeFailedResponse/too_many_requests === RUN TestDecodeFailedResponse/too_large_entry --- PASS: TestDecodeFailedResponse (0.00s) --- PASS: TestDecodeFailedResponse/internal_error (0.00s) --- PASS: TestDecodeFailedResponse/too_many_requests (0.00s) --- PASS: TestDecodeFailedResponse/too_large_entry (0.00s) === RUN TestResponseRoundtrip === RUN TestResponseRoundtrip/successful_string_response === RUN TestResponseRoundtrip/successful_scalar_response === RUN TestResponseRoundtrip/successful_instant_response === RUN TestResponseRoundtrip/successful_range_response === RUN TestResponseRoundtrip/successful_empty_matrix_response === RUN TestResponseRoundtrip/error_response --- PASS: TestResponseRoundtrip (0.01s) --- PASS: TestResponseRoundtrip/successful_string_response (0.00s) --- PASS: TestResponseRoundtrip/successful_scalar_response (0.00s) --- PASS: TestResponseRoundtrip/successful_instant_response (0.00s) --- PASS: TestResponseRoundtrip/successful_range_response (0.00s) --- PASS: TestResponseRoundtrip/successful_empty_matrix_response (0.00s) --- PASS: TestResponseRoundtrip/error_response (0.00s) === RUN TestMergeAPIResponses === RUN TestMergeAPIResponses/No_responses_shouldn't_panic_and_return_a_non-null_result_and_result_type. === RUN TestMergeAPIResponses/A_single_empty_response_shouldn't_panic. === RUN TestMergeAPIResponses/Multiple_empty_responses_shouldn't_panic. === RUN TestMergeAPIResponses/Basic_merging_of_two_responses. === RUN TestMergeAPIResponses/Merging_of_responses_when_labels_are_in_different_order. === RUN TestMergeAPIResponses/Merging_of_samples_where_there_is_single_overlap. === RUN TestMergeAPIResponses/Merging_of_samples_where_there_is_multiple_partial_overlaps. === RUN TestMergeAPIResponses/Merging_of_samples_where_there_is_complete_overlap. === RUN TestMergeAPIResponses/shouldn't_merge_unsuccessful_responses === RUN TestMergeAPIResponses/shouldn't_merge_nil_data === RUN TestMergeAPIResponses/shouldn't_merge_non-matrix_data --- PASS: TestMergeAPIResponses (0.00s) --- PASS: TestMergeAPIResponses/No_responses_shouldn't_panic_and_return_a_non-null_result_and_result_type. (0.00s) --- PASS: TestMergeAPIResponses/A_single_empty_response_shouldn't_panic. (0.00s) --- PASS: TestMergeAPIResponses/Multiple_empty_responses_shouldn't_panic. (0.00s) --- PASS: TestMergeAPIResponses/Basic_merging_of_two_responses. (0.00s) --- PASS: TestMergeAPIResponses/Merging_of_responses_when_labels_are_in_different_order. (0.00s) --- PASS: TestMergeAPIResponses/Merging_of_samples_where_there_is_single_overlap. (0.00s) --- PASS: TestMergeAPIResponses/Merging_of_samples_where_there_is_multiple_partial_overlaps. (0.00s) --- PASS: TestMergeAPIResponses/Merging_of_samples_where_there_is_complete_overlap. (0.00s) --- PASS: TestMergeAPIResponses/shouldn't_merge_unsuccessful_responses (0.00s) --- PASS: TestMergeAPIResponses/shouldn't_merge_nil_data (0.00s) --- PASS: TestMergeAPIResponses/shouldn't_merge_non-matrix_data (0.00s) === RUN Test_DecodeOptions === RUN Test_DecodeOptions/default === PAUSE Test_DecodeOptions/default === RUN Test_DecodeOptions/disable_cache === PAUSE Test_DecodeOptions/disable_cache === RUN Test_DecodeOptions/custom_sharding === PAUSE Test_DecodeOptions/custom_sharding === RUN Test_DecodeOptions/disable_sharding === PAUSE Test_DecodeOptions/disable_sharding === RUN Test_DecodeOptions/custom_instant_query_splitting === PAUSE Test_DecodeOptions/custom_instant_query_splitting === RUN Test_DecodeOptions/disable_instant_query_splitting === PAUSE Test_DecodeOptions/disable_instant_query_splitting === CONT Test_DecodeOptions/default === CONT Test_DecodeOptions/disable_instant_query_splitting === CONT Test_DecodeOptions/custom_instant_query_splitting === CONT Test_DecodeOptions/disable_sharding === CONT Test_DecodeOptions/custom_sharding === CONT Test_DecodeOptions/disable_cache --- PASS: Test_DecodeOptions (0.00s) --- PASS: Test_DecodeOptions/default (0.00s) --- PASS: Test_DecodeOptions/disable_instant_query_splitting (0.00s) --- PASS: Test_DecodeOptions/custom_instant_query_splitting (0.00s) --- PASS: Test_DecodeOptions/disable_sharding (0.00s) --- PASS: Test_DecodeOptions/custom_sharding (0.00s) --- PASS: Test_DecodeOptions/disable_cache (0.00s) === RUN TestLimitsMiddleware_MaxQueryLookback === RUN TestLimitsMiddleware_MaxQueryLookback/should_not_manipulate_time_range_if_max_lookback_is_disabled === RUN TestLimitsMiddleware_MaxQueryLookback/should_not_manipulate_time_range_for_a_query_on_short_time_range === RUN TestLimitsMiddleware_MaxQueryLookback/should_not_manipulate_a_query_on_large_time_range_close_to_the_limit === RUN TestLimitsMiddleware_MaxQueryLookback/should_manipulate_a_query_on_large_time_range_over_the_limit === RUN TestLimitsMiddleware_MaxQueryLookback/should_skip_executing_a_query_outside_the_allowed_time_range === RUN TestLimitsMiddleware_MaxQueryLookback/should_manipulate_a_query_where_maxQueryLookback_is_past_the_retention_period --- PASS: TestLimitsMiddleware_MaxQueryLookback (0.01s) --- PASS: TestLimitsMiddleware_MaxQueryLookback/should_not_manipulate_time_range_if_max_lookback_is_disabled (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLookback/should_not_manipulate_time_range_for_a_query_on_short_time_range (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLookback/should_not_manipulate_a_query_on_large_time_range_close_to_the_limit (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLookback/should_manipulate_a_query_on_large_time_range_over_the_limit (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLookback/should_skip_executing_a_query_outside_the_allowed_time_range (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLookback/should_manipulate_a_query_where_maxQueryLookback_is_past_the_retention_period (0.00s) === RUN TestLimitsMiddleware_MaxQueryLength === RUN TestLimitsMiddleware_MaxQueryLength/should_succeed_on_a_query_on_short_time_range,_ending_now === RUN TestLimitsMiddleware_MaxQueryLength/should_succeed_on_a_query_on_short_time_range,_ending_in_the_past === RUN TestLimitsMiddleware_MaxQueryLength/should_succeed_on_a_query_on_large_time_range_close_to_the_limit,_ending_now === RUN TestLimitsMiddleware_MaxQueryLength/should_fail_on_a_query_on_large_time_range_over_the_limit,_ending_now === RUN TestLimitsMiddleware_MaxQueryLength/should_fail_on_a_query_on_large_time_range_over_the_limit,_ending_in_the_past === RUN TestLimitsMiddleware_MaxQueryLength/should_succeed_if_total_query_length_is_higher_than_query_length_limit === RUN TestLimitsMiddleware_MaxQueryLength/should_skip_validation_if_max_length_is_disabled --- PASS: TestLimitsMiddleware_MaxQueryLength (0.01s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_succeed_on_a_query_on_short_time_range,_ending_now (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_succeed_on_a_query_on_short_time_range,_ending_in_the_past (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_succeed_on_a_query_on_large_time_range_close_to_the_limit,_ending_now (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_fail_on_a_query_on_large_time_range_over_the_limit,_ending_now (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_fail_on_a_query_on_large_time_range_over_the_limit,_ending_in_the_past (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_succeed_if_total_query_length_is_higher_than_query_length_limit (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_skip_validation_if_max_length_is_disabled (0.01s) === RUN TestLimitsMiddleware_CreationGracePeriod === RUN TestLimitsMiddleware_CreationGracePeriod/should_manipulate_time_range_for_a_query_over_now_+_creation_grace_period === RUN TestLimitsMiddleware_CreationGracePeriod/should_not_manipulate_time_range_if_creation_grace_period_is_disabled === RUN TestLimitsMiddleware_CreationGracePeriod/should_not_manipulate_time_range_for_a_query_in_now_+_creation_grace_period --- PASS: TestLimitsMiddleware_CreationGracePeriod (0.00s) --- PASS: TestLimitsMiddleware_CreationGracePeriod/should_manipulate_time_range_for_a_query_over_now_+_creation_grace_period (0.00s) --- PASS: TestLimitsMiddleware_CreationGracePeriod/should_not_manipulate_time_range_if_creation_grace_period_is_disabled (0.00s) --- PASS: TestLimitsMiddleware_CreationGracePeriod/should_not_manipulate_time_range_for_a_query_in_now_+_creation_grace_period (0.00s) === RUN TestLimitedRoundTripper_MaxQueryParallelism --- PASS: TestLimitedRoundTripper_MaxQueryParallelism (0.23s) === RUN TestLimitedRoundTripper_MaxQueryParallelismLateScheduling --- PASS: TestLimitedRoundTripper_MaxQueryParallelismLateScheduling (0.02s) === RUN TestLimitedRoundTripper_OriginalRequestContextCancellation --- PASS: TestLimitedRoundTripper_OriginalRequestContextCancellation (0.10s) === RUN TestQueryShardingCorrectness === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2 === PAUSE TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2 === RUN TestQueryShardingCorrectness/label_replace === PAUSE TestQueryShardingCorrectness/label_replace === RUN TestQueryShardingCorrectness/stddev() === PAUSE TestQueryShardingCorrectness/stddev() === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month() === PAUSE TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month() === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without' === PAUSE TestQueryShardingCorrectness/histogram_quantile()_grouping_'without' === RUN TestQueryShardingCorrectness/max()_grouping_'by' === PAUSE TestQueryShardingCorrectness/max()_grouping_'by' === RUN TestQueryShardingCorrectness/count()_no_grouping === PAUSE TestQueryShardingCorrectness/count()_no_grouping === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand === PAUSE TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without' === PAUSE TestQueryShardingCorrectness/sum(rate())_grouping_'without' === RUN TestQueryShardingCorrectness/and === PAUSE TestQueryShardingCorrectness/and === RUN TestQueryShardingCorrectness/double_subquery_deriv === PAUSE TestQueryShardingCorrectness/double_subquery_deriv === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant === PAUSE TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant === RUN TestQueryShardingCorrectness/string_literal === PAUSE TestQueryShardingCorrectness/string_literal === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""}) === PAUSE TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""}) === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate()) === PAUSE TestQueryShardingCorrectness/sum(rate())_>_avg(rate()) === RUN TestQueryShardingCorrectness/triple_subquery === PAUSE TestQueryShardingCorrectness/triple_subquery === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_avg === PAUSE TestQueryShardingCorrectness/outer_subquery_on_top_of_avg === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series === PAUSE TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series === RUN TestQueryShardingCorrectness/avg()_no_grouping === PAUSE TestQueryShardingCorrectness/avg()_no_grouping === RUN TestQueryShardingCorrectness/avg()_grouping_'by' === PAUSE TestQueryShardingCorrectness/avg()_grouping_'by' === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter)) === PAUSE TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter)) === RUN TestQueryShardingCorrectness/month()_>=_1_and_month() === PAUSE TestQueryShardingCorrectness/month()_>=_1_and_month() === RUN TestQueryShardingCorrectness/max()_grouping_'without' === PAUSE TestQueryShardingCorrectness/max()_grouping_'without' === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1) === PAUSE TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1) === RUN TestQueryShardingCorrectness/nested_count() === PAUSE TestQueryShardingCorrectness/nested_count() === RUN TestQueryShardingCorrectness/@_modifier === PAUSE TestQueryShardingCorrectness/@_modifier === RUN TestQueryShardingCorrectness/@_modifier_and_offset === PAUSE TestQueryShardingCorrectness/@_modifier_and_offset === RUN TestQueryShardingCorrectness/scalar(single_metric) === PAUSE TestQueryShardingCorrectness/scalar(single_metric) === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1 === PAUSE TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1 === RUN TestQueryShardingCorrectness/0_<_bool_1 === PAUSE TestQueryShardingCorrectness/0_<_bool_1 === RUN TestQueryShardingCorrectness/count()_grouping_'by' === PAUSE TestQueryShardingCorrectness/count()_grouping_'by' === RUN TestQueryShardingCorrectness/avg()_grouping_'without' === PAUSE TestQueryShardingCorrectness/avg()_grouping_'without' === RUN TestQueryShardingCorrectness/sum(max_over_time()) === PAUSE TestQueryShardingCorrectness/sum(max_over_time()) === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar === PAUSE TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_sum === PAUSE TestQueryShardingCorrectness/outer_subquery_on_top_of_sum === RUN TestQueryShardingCorrectness/vector() === PAUSE TestQueryShardingCorrectness/vector() === RUN TestQueryShardingCorrectness/vector(1) === PAUSE TestQueryShardingCorrectness/vector(1) === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m])) === PAUSE TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m])) === RUN TestQueryShardingCorrectness/sum()_no_grouping === PAUSE TestQueryShardingCorrectness/sum()_no_grouping === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping === PAUSE TestQueryShardingCorrectness/sum(rate())_no_grouping === RUN TestQueryShardingCorrectness/sum(avg_over_time()) === PAUSE TestQueryShardingCorrectness/sum(avg_over_time()) === RUN TestQueryShardingCorrectness/or === PAUSE TestQueryShardingCorrectness/or === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr === PAUSE TestQueryShardingCorrectness/subquery_min_over_time_with_aggr === RUN TestQueryShardingCorrectness/bottomk() === PAUSE TestQueryShardingCorrectness/bottomk() === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation === PAUSE TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation === RUN TestQueryShardingCorrectness/sum()_grouping_'by' === PAUSE TestQueryShardingCorrectness/sum()_grouping_'by' === RUN TestQueryShardingCorrectness/min()_grouping_'by' === PAUSE TestQueryShardingCorrectness/min()_grouping_'by' === RUN TestQueryShardingCorrectness/scalar(aggregation) === PAUSE TestQueryShardingCorrectness/scalar(aggregation) === RUN TestQueryShardingCorrectness/month(sum(metric_counter)) === PAUSE TestQueryShardingCorrectness/month(sum(metric_counter)) === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1 === PAUSE TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by' === PAUSE TestQueryShardingCorrectness/sum(rate())_grouping_'by' === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series === PAUSE TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series === RUN TestQueryShardingCorrectness/max()_no_grouping === PAUSE TestQueryShardingCorrectness/max()_no_grouping === RUN TestQueryShardingCorrectness/subquery_max === PAUSE TestQueryShardingCorrectness/subquery_max === RUN TestQueryShardingCorrectness/subquery_min === PAUSE TestQueryShardingCorrectness/subquery_min === RUN TestQueryShardingCorrectness/count()_grouping_'without' === PAUSE TestQueryShardingCorrectness/count()_grouping_'without' === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children === PAUSE TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children === RUN TestQueryShardingCorrectness/time() === PAUSE TestQueryShardingCorrectness/time() === RUN TestQueryShardingCorrectness/sum()_offset === PAUSE TestQueryShardingCorrectness/sum()_offset === RUN TestQueryShardingCorrectness/sum()_negative_offset === PAUSE TestQueryShardingCorrectness/sum()_negative_offset === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le === PAUSE TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar === PAUSE TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand === PAUSE TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping === PAUSE TestQueryShardingCorrectness/histogram_quantile()_no_grouping === RUN TestQueryShardingCorrectness/sum(min_over_time()) === PAUSE TestQueryShardingCorrectness/sum(min_over_time()) === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1) === PAUSE TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1) === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by' === PAUSE TestQueryShardingCorrectness/histogram_quantile()_grouping_'by' === RUN TestQueryShardingCorrectness/sum(count()) === PAUSE TestQueryShardingCorrectness/sum(count()) === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order === PAUSE TestQueryShardingCorrectness/query_with_sort()_expects_specific_order === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand === PAUSE TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand === RUN TestQueryShardingCorrectness/stdvar() === PAUSE TestQueryShardingCorrectness/stdvar() === RUN TestQueryShardingCorrectness/sum({__name__!=""}) === PAUSE TestQueryShardingCorrectness/sum({__name__!=""}) === RUN TestQueryShardingCorrectness/min()_grouping_'without' === PAUSE TestQueryShardingCorrectness/min()_grouping_'without' === RUN TestQueryShardingCorrectness/sum_of_subquery_min === PAUSE TestQueryShardingCorrectness/sum_of_subquery_min === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation === PAUSE TestQueryShardingCorrectness/histogram_quantile_without_aggregation === RUN TestQueryShardingCorrectness/sum()_grouping_'without' === PAUSE TestQueryShardingCorrectness/sum()_grouping_'without' === RUN TestQueryShardingCorrectness/min()_no_grouping === PAUSE TestQueryShardingCorrectness/min()_no_grouping === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2 === PAUSE TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month() === PAUSE TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month() === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset === PAUSE TestQueryShardingCorrectness/@_modifier_and_negative_offset === RUN TestQueryShardingCorrectness/label_join === PAUSE TestQueryShardingCorrectness/label_join === RUN TestQueryShardingCorrectness/topk() === PAUSE TestQueryShardingCorrectness/topk() === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1) === PAUSE TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1) === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1) === PAUSE TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1) === CONT TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2 === CONT TestQueryShardingCorrectness/subquery_max === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest === CONT TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand === CONT TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/max()_no_grouping === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/sum(rate())_grouping_'by' === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/month(sum(metric_counter)) === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/scalar(aggregation) === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest === CONT TestQueryShardingCorrectness/min()_grouping_'by' === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum()_grouping_'by' === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1) === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1) === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/bottomk() === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/topk() === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/subquery_min_over_time_with_aggr === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/label_join === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/@_modifier_and_negative_offset === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month() === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/or === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum(avg_over_time()) === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest === CONT TestQueryShardingCorrectness/sum(rate())_no_grouping === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/sum()_no_grouping === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/query_with_sort()_expects_specific_order === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum(count()) === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest === CONT TestQueryShardingCorrectness/histogram_quantile()_grouping_'by' === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1) === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/sum(min_over_time()) === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/histogram_quantile()_no_grouping === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum()_negative_offset === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/sum()_offset === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/time() === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/count()_grouping_'without' === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/subquery_min === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest === CONT TestQueryShardingCorrectness/min()_no_grouping === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum()_grouping_'without' === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m])) === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/vector(1) === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/vector() === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/outer_subquery_on_top_of_sum === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/sum(max_over_time()) === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/avg()_grouping_'without' === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/count()_grouping_'by' === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/0_<_bool_1 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/scalar(single_metric) === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/@_modifier_and_offset === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/@_modifier === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/nested_count() === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1) === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/max()_grouping_'without' === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/month()_>=_1_and_month() === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter)) === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/avg()_grouping_'by' === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/avg()_no_grouping === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/outer_subquery_on_top_of_avg === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/triple_subquery === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum(rate())_>_avg(rate()) === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""}) === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/string_literal === RUN TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/double_subquery_deriv === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/and === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum(rate())_grouping_'without' === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/count()_no_grouping === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/max()_grouping_'by' === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/histogram_quantile()_grouping_'without' === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month() === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/stddev() === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/label_replace === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/histogram_quantile_without_aggregation === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/sum_of_subquery_min === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/min()_grouping_'without' === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum({__name__!=""}) === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest === CONT TestQueryShardingCorrectness/stdvar() === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=16 --- PASS: TestQueryShardingCorrectness (0.95s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand (2.30s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest (1.02s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.11s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.22s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.20s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest (1.27s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.20s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.36s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.27s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.24s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping (1.29s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest (0.58s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.07s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.17s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.14s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.16s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest (0.71s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.05s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.24s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.22s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.14s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2 (4.63s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest (0.85s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.17s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.17s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.13s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.22s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest (3.78s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.92s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.62s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.71s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.99s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2 (4.67s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest (0.82s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.11s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.14s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.24s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.23s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest (3.84s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.80s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.65s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.72s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.88s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1 (1.77s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest (0.92s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.15s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.19s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.16s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.26s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest (0.81s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.09s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.09s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.13s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.36s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by' (3.86s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest (0.96s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.06s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.31s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.25s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.27s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest (2.89s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.44s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.40s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.45s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.75s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter)) (2.10s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest (0.69s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.11s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.13s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.11s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.23s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest (1.41s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.27s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.15s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.50s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.29s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by' (1.26s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest (0.59s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.05s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.12s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.16s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.14s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest (0.67s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.08s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.15s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.14s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.23s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation) (2.41s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest (0.97s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.19s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.10s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.23s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.22s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest (1.45s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.14s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.45s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.20s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.23s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by' (2.53s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest (0.70s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.11s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.08s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.22s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.27s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest (1.83s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.34s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.28s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.38s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.56s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1) (2.17s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest (0.69s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.08s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.13s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.17s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.27s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest (1.48s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.16s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.19s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.18s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.30s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1) (0.19s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest (0.12s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.04s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.02s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.03s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest (0.07s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.04s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/bottomk() (2.31s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest (0.49s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.04s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.13s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.13s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.10s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest (1.81s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.37s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.24s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.25s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.27s) --- PASS: TestQueryShardingCorrectness/topk() (4.01s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest (0.36s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.09s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.04s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.09s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.07s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest (3.64s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.46s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (1.26s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.62s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.54s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series (18.34s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest (1.75s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.56s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.28s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.18s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.65s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest (16.58s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (4.97s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (3.60s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (2.62s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (2.03s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr (2.90s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest (1.10s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.35s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.17s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.25s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.15s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest (1.80s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.28s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.67s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.29s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.32s) --- PASS: TestQueryShardingCorrectness/label_join (1.22s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest (0.32s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.06s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.09s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.11s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest (0.91s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.18s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.15s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.13s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.23s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset (1.70s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest (0.79s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.14s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.13s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.24s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.17s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest (0.91s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.16s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.11s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.24s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.31s) --- PASS: TestQueryShardingCorrectness/or (1.33s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest (0.52s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.05s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.12s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.28s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest (0.81s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.07s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.13s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.14s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.25s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month() (2.43s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest (0.50s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.06s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.09s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.11s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.14s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest (1.93s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.49s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.25s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.30s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.22s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping (2.17s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest (0.52s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.11s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.16s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.16s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest (1.65s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.40s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.25s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.19s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.31s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time()) (3.74s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest (0.57s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.08s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.10s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.12s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.17s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest (3.15s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.55s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.45s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.57s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.84s) --- PASS: TestQueryShardingCorrectness/subquery_max (29.40s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest (3.66s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.76s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.51s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (1.03s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (1.07s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest (25.73s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (8.06s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (7.23s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (3.48s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (5.58s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping (1.68s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest (0.48s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.04s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.05s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.19s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.15s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest (1.19s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.64s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.12s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.13s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.13s) --- PASS: TestQueryShardingCorrectness/sum(count()) (2.51s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest (0.39s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.04s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.04s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.07s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.20s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest (2.11s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.45s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.46s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.28s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.64s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1) (1.55s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest (0.77s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.06s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.10s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.18s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.31s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest (0.79s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.14s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.15s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.15s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.20s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order (9.36s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest (1.90s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.23s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.26s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.49s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.66s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest (7.46s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (1.65s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (1.01s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (1.59s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.98s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time()) (5.28s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest (0.91s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.12s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.16s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.28s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.25s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest (4.37s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.83s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.81s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (1.08s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (1.24s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping (0.35s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest (0.14s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.04s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.03s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest (0.20s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.06s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.06s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.03s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand (1.48s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest (0.50s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.08s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.06s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.10s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.16s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest (0.98s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.20s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.14s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.14s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.26s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by' (17.44s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest (3.28s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (1.05s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.45s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.67s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.66s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest (14.16s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (2.91s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (2.07s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (2.64s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (2.24s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar (8.02s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest (0.50s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.13s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.08s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.09s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.15s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest (7.52s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (2.38s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (1.34s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.87s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.50s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset (1.04s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest (0.39s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.07s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.04s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.09s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.14s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest (0.65s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.08s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.11s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.10s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.19s) --- PASS: TestQueryShardingCorrectness/time() (0.17s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest (0.02s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest (0.15s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.03s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.05s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.04s) --- PASS: TestQueryShardingCorrectness/sum()_offset (1.43s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest (0.42s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.05s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.11s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.09s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.10s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest (1.01s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.18s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.15s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.24s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.29s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children (3.29s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest (1.23s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.21s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.21s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.13s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.22s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest (2.05s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.34s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.70s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.25s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.46s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without' (3.02s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest (0.49s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.07s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.11s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.12s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.15s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest (2.53s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.42s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.52s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.63s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.67s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping (1.19s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest (0.51s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.05s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.11s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.14s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.20s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest (0.68s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.10s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.09s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.19s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.23s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le (14.21s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest (3.82s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.52s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.32s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.74s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (1.25s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest (10.39s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (1.84s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (1.42s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.93s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (1.18s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without' (5.39s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest (0.79s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.15s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.16s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.16s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.23s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest (4.60s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.63s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.49s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.97s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (1.63s) --- PASS: TestQueryShardingCorrectness/vector(1) (0.09s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest (0.03s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest (0.06s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/vector() (0.07s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest (0.04s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.02s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest (0.03s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_sum (0.46s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest (0.46s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.07s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.18s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.09s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.07s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation (51.63s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest (3.99s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (1.09s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.86s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.59s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.82s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest (47.64s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (9.03s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (10.34s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (11.87s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (9.46s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m])) (12.94s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest (3.68s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.90s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.39s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (1.01s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.74s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest (9.25s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (2.28s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (1.28s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (1.75s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (1.46s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time()) (6.04s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest (1.40s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.08s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.20s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.39s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.63s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest (4.64s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.63s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.84s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.81s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.87s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by' (2.40s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest (0.56s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.06s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.10s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.19s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.17s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest (1.84s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.18s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.60s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.27s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.48s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1 (0.31s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest (0.12s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.04s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.04s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest (0.19s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.04s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.04s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.03s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.06s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without' (6.51s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest (1.04s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.13s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.19s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.28s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.41s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest (5.46s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (1.14s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.96s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (1.34s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (1.46s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric) (0.29s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest (0.13s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.02s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest (0.15s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.04s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.03s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.03s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1 (1.66s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest (0.54s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.25s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.10s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.02s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.14s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest (1.12s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.28s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.22s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.15s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.17s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset (3.45s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest (1.04s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.15s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.17s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.27s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.33s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest (2.36s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.59s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.46s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.39s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.54s) --- PASS: TestQueryShardingCorrectness/@_modifier (5.19s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest (2.31s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.37s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.34s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.54s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.59s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest (2.86s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.50s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.36s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.50s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.85s) --- PASS: TestQueryShardingCorrectness/nested_count() (5.03s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest (1.05s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.09s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.10s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.20s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.25s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest (3.97s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.36s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.84s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.78s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (1.27s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without' (1.13s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest (0.50s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.04s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.10s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.16s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.16s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest (0.61s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.10s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.11s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.11s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.23s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month() (0.42s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest (0.27s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.05s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.03s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.12s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.05s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest (0.15s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.05s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/subquery_min (33.87s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest (3.46s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.75s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.73s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.91s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.87s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest (30.41s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (5.67s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (7.92s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (7.85s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (8.50s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by' (3.32s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest (0.91s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.09s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.18s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.27s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.27s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest (2.41s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.31s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.31s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.35s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (1.23s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping (2.31s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest (0.66s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.10s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.15s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.10s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.26s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest (1.65s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.49s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.18s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.28s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.49s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter)) (10.23s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest (1.54s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.14s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.17s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.24s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.34s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest (8.69s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (1.38s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (1.14s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.83s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.79s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar (36.22s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest (1.97s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.38s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.34s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.50s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.54s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest (34.25s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (9.13s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (8.37s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (7.77s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (7.23s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_avg (0.70s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest (0.70s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.13s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.16s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.28s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.08s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series (4.88s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest (0.33s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.06s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.09s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.12s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest (4.54s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.92s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.99s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (1.45s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.97s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate()) (6.85s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest (1.48s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.14s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.24s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.23s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.74s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest (5.35s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (1.12s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.66s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.99s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.74s) --- PASS: TestQueryShardingCorrectness/string_literal (0.02s) --- PASS: TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest (0.02s) --- PASS: TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""}) (11.69s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest (2.70s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.68s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.34s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.37s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.94s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest (8.98s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (1.47s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (1.34s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.82s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (1.03s) --- PASS: TestQueryShardingCorrectness/triple_subquery (30.39s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest (5.33s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (1.02s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.72s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (1.37s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (1.68s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest (25.06s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (6.86s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (6.11s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (5.69s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (4.37s) --- PASS: TestQueryShardingCorrectness/and (3.50s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest (0.94s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.11s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.09s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.15s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.34s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest (2.56s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.72s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.31s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.64s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.62s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv (24.48s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest (3.33s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.70s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.39s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.91s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.97s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest (21.15s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (4.76s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (6.15s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (3.45s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (6.02s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without' (5.46s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest (0.48s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.05s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.08s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.08s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.18s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest (4.97s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.68s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.84s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.99s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (1.65s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping (2.19s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest (0.72s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.09s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.10s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.17s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.28s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest (1.47s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.26s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.14s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.17s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.26s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by' (0.89s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest (0.41s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.04s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.06s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.07s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.19s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest (0.48s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.06s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.07s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.12s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.16s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant (49.28s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest (2.46s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.25s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.91s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.51s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.66s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest (46.83s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (8.43s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (8.67s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (11.10s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (13.58s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without' (14.01s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest (3.15s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (1.20s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.53s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.41s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.61s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest (10.85s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (1.88s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (1.09s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (1.04s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (1.33s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month() (0.14s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest (0.06s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest (0.07s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/stddev() (1.32s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest (0.20s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.04s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.04s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.03s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest (1.12s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.17s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.20s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.24s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.25s) --- PASS: TestQueryShardingCorrectness/label_replace (1.47s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest (0.36s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.04s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.05s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.11s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.13s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest (1.11s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.17s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.15s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.18s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.41s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min (4.21s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest (1.00s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.13s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.15s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.10s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.17s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest (3.21s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (1.26s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.37s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.46s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.46s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without' (0.71s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest (0.25s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.03s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.06s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.09s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest (0.45s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.06s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.08s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.10s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.17s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation (7.83s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest (0.80s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.19s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.23s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.14s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.08s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest (7.02s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (2.31s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (1.32s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (1.19s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (1.63s) --- PASS: TestQueryShardingCorrectness/stdvar() (1.61s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest (0.23s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.06s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.04s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.04s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.04s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest (1.39s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.20s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.21s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.29s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.44s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""}) (7.40s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest (1.95s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.25s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.34s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.33s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.61s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest (5.45s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (1.08s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.64s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (1.02s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.84s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand (44.67s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest (3.52s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.48s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (1.29s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.60s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.83s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest (41.14s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (12.86s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (11.28s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (6.31s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (5.00s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1) (99.02s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest (5.42s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (1.40s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (1.47s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.98s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (1.23s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest (93.60s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (21.99s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (23.73s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (23.97s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (11.33s) === RUN TestQueryshardingDeterminism --- PASS: TestQueryshardingDeterminism (0.79s) === RUN TestQuerySharding_FunctionCorrectness === RUN TestQuerySharding_FunctionCorrectness/(abs(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(abs(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(abs(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(abs(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(abs(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(avg_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(avg_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(avg_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(avg_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(avg_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(ceil(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(ceil(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(ceil(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(ceil(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(ceil(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(changes(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(changes(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(changes(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(changes(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(changes(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(count_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(count_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(count_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(count_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(count_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(days_in_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(days_in_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(days_in_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(days_in_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(days_in_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(day_of_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(day_of_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(day_of_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(day_of_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(day_of_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(day_of_week(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(day_of_week(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(day_of_week(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(day_of_week(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(day_of_week(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(day_of_year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(day_of_year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(day_of_year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(day_of_year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(day_of_year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(delta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(delta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(delta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(delta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(delta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(deriv(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(deriv(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(deriv(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(deriv(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(deriv(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(exp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(exp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(exp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(exp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(exp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(floor(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(floor(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(floor(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(floor(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(floor(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(hour(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(hour(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(hour(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(hour(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(hour(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(idelta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(idelta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(idelta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(idelta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(idelta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(increase(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(increase(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(increase(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(increase(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(increase(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(irate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(irate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(irate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(irate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(irate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(ln(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(ln(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(ln(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(ln(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(ln(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(log10(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(log10(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(log10(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(log10(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(log10(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(log2(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(log2(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(log2(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(log2(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(log2(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(max_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(max_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(max_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(max_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(max_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(min_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(min_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(min_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(min_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(min_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(minute(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(minute(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(minute(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(minute(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(minute(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(rate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(rate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(rate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(rate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(rate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(resets(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(resets(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(resets(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(resets(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(resets(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(sort(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(sort(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sort(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(sort(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(sort(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(sort_desc(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(sort_desc(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sort_desc(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(sort_desc(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(sort_desc(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(sqrt(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(sqrt(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sqrt(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(sqrt(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(sqrt(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(deg(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(deg(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(deg(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(deg(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(deg(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(asinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(asinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(asinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(asinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(asinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(rad(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(rad(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(rad(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(rad(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(rad(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(cosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(cosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(cosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(cosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(cosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(atan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(atan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(atan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(atan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(atan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(atanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(atanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(atanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(atanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(atanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(asin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(asin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(asin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(asin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(asin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(sinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(sinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(sinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(sinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(cos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(cos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(cos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(cos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(cos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(acosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(acosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(acosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(acosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(acosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(sin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(sin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(sin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(sin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(tanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(tanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(tanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(tanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(tanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(tan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(tan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(tan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(tan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(tan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(acos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(acos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(acos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(acos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(acos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(stddev_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(stddev_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(stddev_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(stddev_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(stddev_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(stdvar_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(stdvar_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(stdvar_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(stdvar_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(stdvar_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(sum_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(sum_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sum_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(sum_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(sum_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(last_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(last_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(last_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(last_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(last_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(present_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(present_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(present_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(present_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(present_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(quantile_over_time(0.5,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(quantile_over_time(0.5,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(quantile_over_time(0.5,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(quantile_over_time(0.5,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(quantile_over_time(0.5,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(quantile_over_time(0.99,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(quantile_over_time(0.99,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(quantile_over_time(0.99,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(quantile_over_time(0.99,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(quantile_over_time(0.99,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(timestamp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(timestamp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(timestamp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(timestamp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(timestamp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(sgn(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(sgn(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sgn(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(sgn(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(sgn(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(clamp(bar1{},5,10)) === RUN TestQuerySharding_FunctionCorrectness/sum(clamp(bar1{},5,10)) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(clamp(bar1{},5,10)) === RUN TestQuerySharding_FunctionCorrectness/count(clamp(bar1{},5,10)) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(clamp(bar1{},5,10)) === RUN TestQuerySharding_FunctionCorrectness/(clamp_max(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/sum(clamp_max(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(clamp_max(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/count(clamp_max(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(clamp_max(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/(clamp_min(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/sum(clamp_min(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(clamp_min(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/count(clamp_min(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(clamp_min(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/(predict_linear(bar1{}[1m],1)) === RUN TestQuerySharding_FunctionCorrectness/sum(predict_linear(bar1{}[1m],1)) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(predict_linear(bar1{}[1m],1)) === RUN TestQuerySharding_FunctionCorrectness/count(predict_linear(bar1{}[1m],1)) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(predict_linear(bar1{}[1m],1)) === RUN TestQuerySharding_FunctionCorrectness/(round(bar1{},20)) === RUN TestQuerySharding_FunctionCorrectness/sum(round(bar1{},20)) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(round(bar1{},20)) === RUN TestQuerySharding_FunctionCorrectness/count(round(bar1{},20)) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(round(bar1{},20)) === RUN TestQuerySharding_FunctionCorrectness/(holt_winters(bar1{}[1m],0.5,0.7)) === RUN TestQuerySharding_FunctionCorrectness/sum(holt_winters(bar1{}[1m],0.5,0.7)) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(holt_winters(bar1{}[1m],0.5,0.7)) === RUN TestQuerySharding_FunctionCorrectness/count(holt_winters(bar1{}[1m],0.5,0.7)) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(holt_winters(bar1{}[1m],0.5,0.7)) === RUN TestQuerySharding_FunctionCorrectness/(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) === RUN TestQuerySharding_FunctionCorrectness/sum(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) === RUN TestQuerySharding_FunctionCorrectness/count(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) === RUN TestQuerySharding_FunctionCorrectness/(label_join(bar1{},"fuzz",",","foo","bar")) === RUN TestQuerySharding_FunctionCorrectness/sum(label_join(bar1{},"fuzz",",","foo","bar")) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(label_join(bar1{},"fuzz",",","foo","bar")) === RUN TestQuerySharding_FunctionCorrectness/count(label_join(bar1{},"fuzz",",","foo","bar")) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(label_join(bar1{},"fuzz",",","foo","bar")) --- PASS: TestQuerySharding_FunctionCorrectness (10.18s) --- PASS: TestQuerySharding_FunctionCorrectness/(abs(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(abs(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(abs(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(abs(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(abs(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(avg_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(avg_over_time(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(avg_over_time(bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count(avg_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(avg_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(ceil(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(ceil(bar1{})) (0.06s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(ceil(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/count(ceil(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(ceil(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/(changes(bar1{}[1m])) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(changes(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(changes(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(changes(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(changes(bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(count_over_time(bar1{}[1m])) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(count_over_time(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(count_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(count_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(count_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(days_in_month(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(days_in_month(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(days_in_month(bar1{})) (0.06s) --- PASS: TestQuerySharding_FunctionCorrectness/count(days_in_month(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(days_in_month(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/(day_of_month(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(day_of_month(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(day_of_month(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/count(day_of_month(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(day_of_month(bar1{})) (0.06s) --- PASS: TestQuerySharding_FunctionCorrectness/(day_of_week(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(day_of_week(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(day_of_week(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/count(day_of_week(bar1{})) (0.09s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(day_of_week(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/(day_of_year(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(day_of_year(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(day_of_year(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/count(day_of_year(bar1{})) (0.06s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(day_of_year(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/(delta(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(delta(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(delta(bar1{}[1m])) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/count(delta(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(delta(bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(deriv(bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(deriv(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(deriv(bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count(deriv(bar1{}[1m])) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(deriv(bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(exp(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(exp(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(exp(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count(exp(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(exp(bar1{})) (0.08s) --- PASS: TestQuerySharding_FunctionCorrectness/(floor(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(floor(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(floor(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count(floor(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(floor(bar1{})) (0.08s) --- PASS: TestQuerySharding_FunctionCorrectness/(hour(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(hour(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(hour(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(hour(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(hour(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(idelta(bar1{}[1m])) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(idelta(bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(idelta(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(idelta(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(idelta(bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(increase(bar1{}[1m])) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(increase(bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(increase(bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count(increase(bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(increase(bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(irate(bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(irate(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(irate(bar1{}[1m])) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/count(irate(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(irate(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(ln(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(ln(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(ln(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/count(ln(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(ln(bar1{})) (0.10s) --- PASS: TestQuerySharding_FunctionCorrectness/(log10(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(log10(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(log10(bar1{})) (0.06s) --- PASS: TestQuerySharding_FunctionCorrectness/count(log10(bar1{})) (0.08s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(log10(bar1{})) (0.07s) --- PASS: TestQuerySharding_FunctionCorrectness/(log2(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(log2(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(log2(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/count(log2(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(log2(bar1{})) (0.06s) --- PASS: TestQuerySharding_FunctionCorrectness/(max_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(max_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(max_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(max_over_time(bar1{}[1m])) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(max_over_time(bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(min_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(min_over_time(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(min_over_time(bar1{}[1m])) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/count(min_over_time(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(min_over_time(bar1{}[1m])) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/(minute(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(minute(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(minute(bar1{})) (0.06s) --- PASS: TestQuerySharding_FunctionCorrectness/count(minute(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(minute(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(month(bar1{})) (0.09s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(month(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(month(bar1{})) (0.06s) --- PASS: TestQuerySharding_FunctionCorrectness/count(month(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(month(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/(rate(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(rate(bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(rate(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(rate(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(rate(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(resets(bar1{}[1m])) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(resets(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(resets(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(resets(bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(resets(bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(sort(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(sort(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sort(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(sort(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(sort(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(sort_desc(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(sort_desc(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sort_desc(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(sort_desc(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(sort_desc(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/(sqrt(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(sqrt(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sqrt(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count(sqrt(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(sqrt(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(deg(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(deg(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(deg(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count(deg(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(deg(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(asinh(bar1{})) (0.08s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(asinh(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(asinh(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/count(asinh(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(asinh(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(rad(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(rad(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(rad(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(rad(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(rad(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(cosh(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(cosh(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(cosh(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/count(cosh(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(cosh(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(atan(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(atan(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(atan(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(atan(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(atan(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(atanh(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(atanh(bar1{})) (0.08s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(atanh(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count(atanh(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(atanh(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(asin(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(asin(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(asin(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/count(asin(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(asin(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(sinh(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(sinh(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sinh(bar1{})) (0.06s) --- PASS: TestQuerySharding_FunctionCorrectness/count(sinh(bar1{})) (0.06s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(sinh(bar1{})) (0.07s) --- PASS: TestQuerySharding_FunctionCorrectness/(cos(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(cos(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(cos(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/count(cos(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(cos(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(acosh(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(acosh(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(acosh(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/count(acosh(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(acosh(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(sin(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(sin(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sin(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count(sin(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(sin(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(tanh(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(tanh(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(tanh(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count(tanh(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(tanh(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/(tan(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(tan(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(tan(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count(tan(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(tan(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(acos(bar1{})) (0.06s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(acos(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(acos(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(acos(bar1{})) (0.06s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(acos(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(stddev_over_time(bar1{}[1m])) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(stddev_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(stddev_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(stddev_over_time(bar1{}[1m])) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(stddev_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(stdvar_over_time(bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(stdvar_over_time(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(stdvar_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(stdvar_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(stdvar_over_time(bar1{}[1m])) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/(sum_over_time(bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(sum_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sum_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(sum_over_time(bar1{}[1m])) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(sum_over_time(bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(last_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(last_over_time(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(last_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(last_over_time(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(last_over_time(bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(present_over_time(bar1{}[1m])) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(present_over_time(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(present_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(present_over_time(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(present_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(quantile_over_time(0.5,bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(quantile_over_time(0.5,bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(quantile_over_time(0.5,bar1{}[1m])) (0.07s) --- PASS: TestQuerySharding_FunctionCorrectness/count(quantile_over_time(0.5,bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(quantile_over_time(0.5,bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(quantile_over_time(0.99,bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(quantile_over_time(0.99,bar1{}[1m])) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(quantile_over_time(0.99,bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count(quantile_over_time(0.99,bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(quantile_over_time(0.99,bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(timestamp(bar1{})) (0.06s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(timestamp(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(timestamp(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count(timestamp(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(timestamp(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(year(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(year(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(year(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count(year(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(year(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(sgn(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(sgn(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sgn(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(sgn(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(sgn(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(clamp(bar1{},5,10)) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(clamp(bar1{},5,10)) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(clamp(bar1{},5,10)) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(clamp(bar1{},5,10)) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(clamp(bar1{},5,10)) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/(clamp_max(bar1{},5)) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(clamp_max(bar1{},5)) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(clamp_max(bar1{},5)) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count(clamp_max(bar1{},5)) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(clamp_max(bar1{},5)) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/(clamp_min(bar1{},5)) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(clamp_min(bar1{},5)) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(clamp_min(bar1{},5)) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/count(clamp_min(bar1{},5)) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(clamp_min(bar1{},5)) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/(predict_linear(bar1{}[1m],1)) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(predict_linear(bar1{}[1m],1)) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(predict_linear(bar1{}[1m],1)) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/count(predict_linear(bar1{}[1m],1)) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(predict_linear(bar1{}[1m],1)) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(round(bar1{},20)) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(round(bar1{},20)) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(round(bar1{},20)) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/count(round(bar1{},20)) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(round(bar1{},20)) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/(holt_winters(bar1{}[1m],0.5,0.7)) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(holt_winters(bar1{}[1m],0.5,0.7)) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(holt_winters(bar1{}[1m],0.5,0.7)) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/count(holt_winters(bar1{}[1m],0.5,0.7)) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(holt_winters(bar1{}[1m],0.5,0.7)) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/count(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(label_join(bar1{},"fuzz",",","foo","bar")) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(label_join(bar1{},"fuzz",",","foo","bar")) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(label_join(bar1{},"fuzz",",","foo","bar")) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/count(label_join(bar1{},"fuzz",",","foo","bar")) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(label_join(bar1{},"fuzz",",","foo","bar")) (0.05s) === RUN TestQuerySharding_ShouldFallbackToDownstreamHandlerOnMappingFailure --- PASS: TestQuerySharding_ShouldFallbackToDownstreamHandlerOnMappingFailure (0.00s) === RUN TestQuerySharding_ShouldSkipShardingViaOption --- PASS: TestQuerySharding_ShouldSkipShardingViaOption (0.00s) === RUN TestQuerySharding_ShouldOverrideShardingSizeViaOption --- PASS: TestQuerySharding_ShouldOverrideShardingSizeViaOption (0.26s) === RUN TestQuerySharding_ShouldSupportMaxShardedQueries === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/query_is_not_shardable === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_1_shardable_leg === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_4_compactor_shards === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_10_compactor_shards === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_14_compactor_shards === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/single_splitted_query,_query_has_1_shardable_leg === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/single_splitted_query,_query_has_many_shardable_legs === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_no_compactor_shards === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_3_compactor_shards === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_11_compactor_shards === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/query_sharding_is_disabled --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries (0.34s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/query_is_not_shardable (0.00s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_1_shardable_leg (0.02s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_4_compactor_shards (0.03s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_10_compactor_shards (0.03s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_14_compactor_shards (0.02s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/single_splitted_query,_query_has_1_shardable_leg (0.06s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/single_splitted_query,_query_has_many_shardable_legs (0.06s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs (0.01s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_no_compactor_shards (0.03s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_3_compactor_shards (0.02s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_11_compactor_shards (0.04s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/query_sharding_is_disabled (0.00s) === RUN TestQuerySharding_ShouldReturnErrorOnDownstreamHandlerFailure --- PASS: TestQuerySharding_ShouldReturnErrorOnDownstreamHandlerFailure (0.00s) === RUN TestQuerySharding_ShouldReturnErrorInCorrectFormat === RUN TestQuerySharding_ShouldReturnErrorInCorrectFormat/downstream_-_timeout === RUN TestQuerySharding_ShouldReturnErrorInCorrectFormat/sharding_-_sample_limit === RUN TestQuerySharding_ShouldReturnErrorInCorrectFormat/sharding_-_timeout === RUN TestQuerySharding_ShouldReturnErrorInCorrectFormat/downstream_-_storage_internal_error === RUN TestQuerySharding_ShouldReturnErrorInCorrectFormat/downstream_-_storage_prometheus_execution_error --- PASS: TestQuerySharding_ShouldReturnErrorInCorrectFormat (0.69s) --- PASS: TestQuerySharding_ShouldReturnErrorInCorrectFormat/downstream_-_timeout (0.34s) --- PASS: TestQuerySharding_ShouldReturnErrorInCorrectFormat/sharding_-_sample_limit (0.00s) --- PASS: TestQuerySharding_ShouldReturnErrorInCorrectFormat/sharding_-_timeout (0.34s) --- PASS: TestQuerySharding_ShouldReturnErrorInCorrectFormat/downstream_-_storage_internal_error (0.00s) --- PASS: TestQuerySharding_ShouldReturnErrorInCorrectFormat/downstream_-_storage_prometheus_execution_error (0.00s) === RUN TestQuerySharding_EngineErrorMapping --- PASS: TestQuerySharding_EngineErrorMapping (0.13s) === RUN TestQuerySharding_WrapMultipleTime --- PASS: TestQuerySharding_WrapMultipleTime (0.00s) === RUN TestPromqlResultToSampleStreams === RUN TestPromqlResultToSampleStreams/[0] === RUN TestPromqlResultToSampleStreams/[1] === RUN TestPromqlResultToSampleStreams/[2] === RUN TestPromqlResultToSampleStreams/[3] === RUN TestPromqlResultToSampleStreams/[4] --- PASS: TestPromqlResultToSampleStreams (0.00s) --- PASS: TestPromqlResultToSampleStreams/[0] (0.00s) --- PASS: TestPromqlResultToSampleStreams/[1] (0.00s) --- PASS: TestPromqlResultToSampleStreams/[2] (0.00s) --- PASS: TestPromqlResultToSampleStreams/[3] (0.00s) --- PASS: TestPromqlResultToSampleStreams/[4] (0.00s) === RUN TestGenLabelsCorrectness --- PASS: TestGenLabelsCorrectness (0.00s) === RUN TestGenLabelsSize --- PASS: TestGenLabelsSize (0.00s) === RUN TestNewMockShardedqueryable --- PASS: TestNewMockShardedqueryable (0.02s) === RUN TestResultsCacheConfig_Validate === RUN TestResultsCacheConfig_Validate/should_pass_with_memcached_backend === RUN TestResultsCacheConfig_Validate/should_pass_with_default_config === RUN TestResultsCacheConfig_Validate/should_fail_with_invalid_memcached_config === RUN TestResultsCacheConfig_Validate/should_fail_with_unsupported_backend --- PASS: TestResultsCacheConfig_Validate (0.00s) --- PASS: TestResultsCacheConfig_Validate/should_pass_with_memcached_backend (0.00s) --- PASS: TestResultsCacheConfig_Validate/should_pass_with_default_config (0.00s) --- PASS: TestResultsCacheConfig_Validate/should_fail_with_invalid_memcached_config (0.00s) --- PASS: TestResultsCacheConfig_Validate/should_fail_with_unsupported_backend (0.00s) === RUN TestIsRequestCachable === RUN TestIsRequestCachable/@_modifier_on_vector_selector,_before_end,_before_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_vector_selector,_after_end,_before_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_vector_selector,_before_end,_after_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_vector_selector,_after_end,_after_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_vector_selector_with_start()_before_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_vector_selector_with_end()_after_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_matrix_selector,_before_end,_before_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_matrix_selector,_after_end,_before_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_matrix_selector,_before_end,_after_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_matrix_selector,_after_end,_after_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_matrix_selector_with_start()_before_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_matrix_selector_with_end()_after_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_subqueries,_before_end,_before_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_subqueries,_after_end,_before_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_subqueries,_before_end,_after_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_subqueries,_after_end,_after_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_subqueries_with_start()_before_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_subqueries_with_end()_after_maxCacheTime === RUN TestIsRequestCachable/positive_offset_on_vector_selector === RUN TestIsRequestCachable/negative_offset_on_vector_selector === RUN TestIsRequestCachable/positive_offset_on_subqueries === RUN TestIsRequestCachable/negative_offset_on_subqueries === RUN TestIsRequestCachable/request_that_is_step_aligned === RUN TestIsRequestCachable/request_that_is_NOT_step_aligned,_with_cacheStepUnaligned=false === RUN TestIsRequestCachable/request_that_is_NOT_step_aligned --- PASS: TestIsRequestCachable (0.01s) --- PASS: TestIsRequestCachable/@_modifier_on_vector_selector,_before_end,_before_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_vector_selector,_after_end,_before_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_vector_selector,_before_end,_after_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_vector_selector,_after_end,_after_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_vector_selector_with_start()_before_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_vector_selector_with_end()_after_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_matrix_selector,_before_end,_before_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_matrix_selector,_after_end,_before_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_matrix_selector,_before_end,_after_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_matrix_selector,_after_end,_after_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_matrix_selector_with_start()_before_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_matrix_selector_with_end()_after_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_subqueries,_before_end,_before_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_subqueries,_after_end,_before_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_subqueries,_before_end,_after_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_subqueries,_after_end,_after_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_subqueries_with_start()_before_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_subqueries_with_end()_after_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/positive_offset_on_vector_selector (0.00s) --- PASS: TestIsRequestCachable/negative_offset_on_vector_selector (0.00s) --- PASS: TestIsRequestCachable/positive_offset_on_subqueries (0.00s) --- PASS: TestIsRequestCachable/negative_offset_on_subqueries (0.00s) --- PASS: TestIsRequestCachable/request_that_is_step_aligned (0.00s) --- PASS: TestIsRequestCachable/request_that_is_NOT_step_aligned,_with_cacheStepUnaligned=false (0.00s) --- PASS: TestIsRequestCachable/request_that_is_NOT_step_aligned (0.00s) === RUN TestIsResponseCachable === RUN TestIsResponseCachable/does_not_contain_the_cacheControl_header === RUN TestIsResponseCachable/does_contain_the_cacheControl_header_which_has_the_value === RUN TestIsResponseCachable/cacheControl_header_contains_extra_values_but_still_good === RUN TestIsResponseCachable/broken_response === RUN TestIsResponseCachable/nil_headers === RUN TestIsResponseCachable/had_cacheControl_header_but_no_values --- PASS: TestIsResponseCachable (0.00s) --- PASS: TestIsResponseCachable/does_not_contain_the_cacheControl_header (0.00s) --- PASS: TestIsResponseCachable/does_contain_the_cacheControl_header_which_has_the_value (0.00s) --- PASS: TestIsResponseCachable/cacheControl_header_contains_extra_values_but_still_good (0.00s) --- PASS: TestIsResponseCachable/broken_response (0.00s) --- PASS: TestIsResponseCachable/nil_headers (0.00s) --- PASS: TestIsResponseCachable/had_cacheControl_header_but_no_values (0.00s) === RUN TestPartitionCacheExtents === RUN TestPartitionCacheExtents/Test_a_complete_hit. === RUN TestPartitionCacheExtents/Test_with_a_complete_miss. === RUN TestPartitionCacheExtents/Test_a_partial_hit. === RUN TestPartitionCacheExtents/Test_multiple_partial_hits. === RUN TestPartitionCacheExtents/Partial_hits_with_tiny_gap. === RUN TestPartitionCacheExtents/Extent_is_outside_the_range_and_the_request_has_a_single_step_(same_start_and_end). === RUN TestPartitionCacheExtents/Test_when_hit_has_a_large_step_and_only_a_single_sample_extent. === RUN TestPartitionCacheExtents/Start_time_of_all_requests_must_have_the_same_offset_into_the_step. --- PASS: TestPartitionCacheExtents (0.00s) --- PASS: TestPartitionCacheExtents/Test_a_complete_hit. (0.00s) --- PASS: TestPartitionCacheExtents/Test_with_a_complete_miss. (0.00s) --- PASS: TestPartitionCacheExtents/Test_a_partial_hit. (0.00s) --- PASS: TestPartitionCacheExtents/Test_multiple_partial_hits. (0.00s) --- PASS: TestPartitionCacheExtents/Partial_hits_with_tiny_gap. (0.00s) --- PASS: TestPartitionCacheExtents/Extent_is_outside_the_range_and_the_request_has_a_single_step_(same_start_and_end). (0.00s) --- PASS: TestPartitionCacheExtents/Test_when_hit_has_a_large_step_and_only_a_single_sample_extent. (0.00s) --- PASS: TestPartitionCacheExtents/Start_time_of_all_requests_must_have_the_same_offset_into_the_step. (0.00s) === RUN TestConstSplitter_generateCacheKey === PAUSE TestConstSplitter_generateCacheKey === RUN TestRetry === RUN TestRetry/retry_failures === RUN TestRetry/don't_retry_400s === RUN TestRetry/retry_500s === RUN TestRetry/last_error --- PASS: TestRetry (0.00s) --- PASS: TestRetry/retry_failures (0.00s) --- PASS: TestRetry/don't_retry_400s (0.00s) --- PASS: TestRetry/retry_500s (0.00s) --- PASS: TestRetry/last_error (0.00s) === RUN Test_RetryMiddlewareCancel --- PASS: Test_RetryMiddlewareCancel (0.00s) === RUN TestRangeTripperware === RUN TestRangeTripperware/0 === RUN TestRangeTripperware/1 --- PASS: TestRangeTripperware (0.01s) --- PASS: TestRangeTripperware/0 (0.00s) --- PASS: TestRangeTripperware/1 (0.00s) === RUN TestInstantTripperware === RUN TestInstantTripperware/specific_time_happy_case === RUN TestInstantTripperware/specific_time_param_with_form_being_already_parsed === RUN TestInstantTripperware/default_time_param_happy_case === RUN TestInstantTripperware/default_time_param_with_form_being_already_parsed === RUN TestInstantTripperware/post_form_time_param_takes_precedence_over_query_time_param_ --- PASS: TestInstantTripperware (0.07s) --- PASS: TestInstantTripperware/specific_time_happy_case (0.01s) --- PASS: TestInstantTripperware/specific_time_param_with_form_being_already_parsed (0.01s) --- PASS: TestInstantTripperware/default_time_param_happy_case (0.02s) --- PASS: TestInstantTripperware/default_time_param_with_form_being_already_parsed (0.01s) --- PASS: TestInstantTripperware/post_form_time_param_takes_precedence_over_query_time_param_ (0.01s) === RUN TestTripperware_Metrics === RUN TestTripperware_Metrics/start/end_is_aligned_to_step === RUN TestTripperware_Metrics/start/end_is_not_aligned_to_step,_aligning_disabled === RUN TestTripperware_Metrics/start/end_is_not_aligned_to_step,_aligning_enabled --- PASS: TestTripperware_Metrics (0.02s) --- PASS: TestTripperware_Metrics/start/end_is_aligned_to_step (0.01s) --- PASS: TestTripperware_Metrics/start/end_is_not_aligned_to_step,_aligning_disabled (0.00s) --- PASS: TestTripperware_Metrics/start/end_is_not_aligned_to_step,_aligning_enabled (0.01s) === RUN TestShardedQuerier_Select === RUN TestShardedQuerier_Select/errors_non_embedded_query === RUN TestShardedQuerier_Select/replaces_query === RUN TestShardedQuerier_Select/propagates_response_error === RUN TestShardedQuerier_Select/returns_SeriesSet --- PASS: TestShardedQuerier_Select (0.00s) --- PASS: TestShardedQuerier_Select/errors_non_embedded_query (0.00s) --- PASS: TestShardedQuerier_Select/replaces_query (0.00s) --- PASS: TestShardedQuerier_Select/propagates_response_error (0.00s) --- PASS: TestShardedQuerier_Select/returns_SeriesSet (0.00s) === RUN TestShardedQuerier_Select_ShouldConcurrentlyRunEmbeddedQueries --- PASS: TestShardedQuerier_Select_ShouldConcurrentlyRunEmbeddedQueries (0.00s) === RUN TestShardedQueryable_GetResponseHeaders --- PASS: TestShardedQueryable_GetResponseHeaders (0.00s) === RUN TestNewSeriesSetFromEmbeddedQueriesResults === RUN TestNewSeriesSetFromEmbeddedQueriesResults/should_add_a_stale_marker_at_the_end_even_if_if_input_samples_have_no_gaps === RUN TestNewSeriesSetFromEmbeddedQueriesResults/should_add_stale_markers_at_the_beginning_of_each_gap_and_one_at_the_end_of_the_series === RUN TestNewSeriesSetFromEmbeddedQueriesResults/should_not_add_stale_markers_even_if_points_have_gaps_if_hints_is_not_passed === RUN TestNewSeriesSetFromEmbeddedQueriesResults/should_not_add_stale_markers_even_if_points_have_gaps_if_step_==_0 --- PASS: TestNewSeriesSetFromEmbeddedQueriesResults (0.00s) --- PASS: TestNewSeriesSetFromEmbeddedQueriesResults/should_add_a_stale_marker_at_the_end_even_if_if_input_samples_have_no_gaps (0.00s) --- PASS: TestNewSeriesSetFromEmbeddedQueriesResults/should_add_stale_markers_at_the_beginning_of_each_gap_and_one_at_the_end_of_the_series (0.00s) --- PASS: TestNewSeriesSetFromEmbeddedQueriesResults/should_not_add_stale_markers_even_if_points_have_gaps_if_hints_is_not_passed (0.00s) --- PASS: TestNewSeriesSetFromEmbeddedQueriesResults/should_not_add_stale_markers_even_if_points_have_gaps_if_step_==_0 (0.00s) === RUN TestResponseToSamples --- PASS: TestResponseToSamples (0.00s) === RUN TestSplitAndCacheMiddleware_SplitByInterval --- PASS: TestSplitAndCacheMiddleware_SplitByInterval (0.01s) === RUN TestSplitAndCacheMiddleware_ResultsCache --- PASS: TestSplitAndCacheMiddleware_ResultsCache (0.00s) === RUN TestSplitAndCacheMiddleware_ResultsCache_ShouldNotLookupCacheIfStepIsNotAligned --- PASS: TestSplitAndCacheMiddleware_ResultsCache_ShouldNotLookupCacheIfStepIsNotAligned (0.00s) === RUN TestSplitAndCacheMiddleware_ResultsCache_EnabledCachingOfStepUnalignedRequest --- PASS: TestSplitAndCacheMiddleware_ResultsCache_EnabledCachingOfStepUnalignedRequest (0.00s) === RUN TestSplitAndCacheMiddleware_ResultsCache_ShouldNotCacheRequestEarlierThanMaxCacheFreshness === RUN TestSplitAndCacheMiddleware_ResultsCache_ShouldNotCacheRequestEarlierThanMaxCacheFreshness/should_not_cache_a_response_if_query_time_range_is_earlier_than_max_cache_freshness === RUN TestSplitAndCacheMiddleware_ResultsCache_ShouldNotCacheRequestEarlierThanMaxCacheFreshness/should_cache_a_response_up_until_max_cache_freshness_time_ago --- PASS: TestSplitAndCacheMiddleware_ResultsCache_ShouldNotCacheRequestEarlierThanMaxCacheFreshness (0.01s) --- PASS: TestSplitAndCacheMiddleware_ResultsCache_ShouldNotCacheRequestEarlierThanMaxCacheFreshness/should_not_cache_a_response_if_query_time_range_is_earlier_than_max_cache_freshness (0.00s) --- PASS: TestSplitAndCacheMiddleware_ResultsCache_ShouldNotCacheRequestEarlierThanMaxCacheFreshness/should_cache_a_response_up_until_max_cache_freshness_time_ago (0.00s) === RUN TestSplitAndCacheMiddleware_ResultsCacheFuzzy split_and_cache_test.go:617: random generator seed: 1678143957914902967 === RUN TestSplitAndCacheMiddleware_ResultsCacheFuzzy/reduced_query_parallelism_(concurrency:_1) === PAUSE TestSplitAndCacheMiddleware_ResultsCacheFuzzy/reduced_query_parallelism_(concurrency:_1) === RUN TestSplitAndCacheMiddleware_ResultsCacheFuzzy/reduced_query_parallelism_(concurrency:_10) === PAUSE TestSplitAndCacheMiddleware_ResultsCacheFuzzy/reduced_query_parallelism_(concurrency:_10) === RUN TestSplitAndCacheMiddleware_ResultsCacheFuzzy/cache_unaligned_requests_(concurrency:_1) === PAUSE TestSplitAndCacheMiddleware_ResultsCacheFuzzy/cache_unaligned_requests_(concurrency:_1) === RUN TestSplitAndCacheMiddleware_ResultsCacheFuzzy/cache_unaligned_requests_(concurrency:_10) === PAUSE TestSplitAndCacheMiddleware_ResultsCacheFuzzy/cache_unaligned_requests_(concurrency:_10) === RUN TestSplitAndCacheMiddleware_ResultsCacheFuzzy/increased_max_cache_freshness_(concurrency:_1) === PAUSE TestSplitAndCacheMiddleware_ResultsCacheFuzzy/increased_max_cache_freshness_(concurrency:_1) === RUN TestSplitAndCacheMiddleware_ResultsCacheFuzzy/increased_max_cache_freshness_(concurrency:_10) === PAUSE TestSplitAndCacheMiddleware_ResultsCacheFuzzy/increased_max_cache_freshness_(concurrency:_10) === RUN TestSplitAndCacheMiddleware_ResultsCacheFuzzy/split_by_interval_disabled_(concurrency:_1) === PAUSE TestSplitAndCacheMiddleware_ResultsCacheFuzzy/split_by_interval_disabled_(concurrency:_1) === RUN TestSplitAndCacheMiddleware_ResultsCacheFuzzy/split_by_interval_disabled_(concurrency:_10) === PAUSE TestSplitAndCacheMiddleware_ResultsCacheFuzzy/split_by_interval_disabled_(concurrency:_10) === RUN TestSplitAndCacheMiddleware_ResultsCacheFuzzy/results_cache_disabled_(concurrency:_1) === PAUSE TestSplitAndCacheMiddleware_ResultsCacheFuzzy/results_cache_disabled_(concurrency:_1) === RUN TestSplitAndCacheMiddleware_ResultsCacheFuzzy/results_cache_disabled_(concurrency:_10) === PAUSE TestSplitAndCacheMiddleware_ResultsCacheFuzzy/results_cache_disabled_(concurrency:_10) === RUN TestSplitAndCacheMiddleware_ResultsCacheFuzzy/both_split_by_interval_and_results_cache_disabled_(concurrency:_1) === PAUSE TestSplitAndCacheMiddleware_ResultsCacheFuzzy/both_split_by_interval_and_results_cache_disabled_(concurrency:_1) === RUN TestSplitAndCacheMiddleware_ResultsCacheFuzzy/both_split_by_interval_and_results_cache_disabled_(concurrency:_10) === PAUSE TestSplitAndCacheMiddleware_ResultsCacheFuzzy/both_split_by_interval_and_results_cache_disabled_(concurrency:_10) === RUN TestSplitAndCacheMiddleware_ResultsCacheFuzzy/default_config_(concurrency:_1) === PAUSE TestSplitAndCacheMiddleware_ResultsCacheFuzzy/default_config_(concurrency:_1) === RUN TestSplitAndCacheMiddleware_ResultsCacheFuzzy/default_config_(concurrency:_10) === PAUSE TestSplitAndCacheMiddleware_ResultsCacheFuzzy/default_config_(concurrency:_10) === CONT TestSplitAndCacheMiddleware_ResultsCacheFuzzy/reduced_query_parallelism_(concurrency:_1) === CONT TestSplitAndCacheMiddleware_ResultsCacheFuzzy/split_by_interval_disabled_(concurrency:_10) === CONT TestSplitAndCacheMiddleware_ResultsCacheFuzzy/increased_max_cache_freshness_(concurrency:_1) === CONT TestSplitAndCacheMiddleware_ResultsCacheFuzzy/cache_unaligned_requests_(concurrency:_1) === CONT TestSplitAndCacheMiddleware_ResultsCacheFuzzy/cache_unaligned_requests_(concurrency:_10) === CONT TestSplitAndCacheMiddleware_ResultsCacheFuzzy/split_by_interval_disabled_(concurrency:_1) === CONT TestSplitAndCacheMiddleware_ResultsCacheFuzzy/increased_max_cache_freshness_(concurrency:_10) === CONT TestSplitAndCacheMiddleware_ResultsCacheFuzzy/reduced_query_parallelism_(concurrency:_10) === CONT TestSplitAndCacheMiddleware_ResultsCacheFuzzy/results_cache_disabled_(concurrency:_10) === CONT TestSplitAndCacheMiddleware_ResultsCacheFuzzy/default_config_(concurrency:_10) === CONT TestSplitAndCacheMiddleware_ResultsCacheFuzzy/default_config_(concurrency:_1) === CONT TestSplitAndCacheMiddleware_ResultsCacheFuzzy/both_split_by_interval_and_results_cache_disabled_(concurrency:_10) === CONT TestSplitAndCacheMiddleware_ResultsCacheFuzzy/both_split_by_interval_and_results_cache_disabled_(concurrency:_1) === CONT TestSplitAndCacheMiddleware_ResultsCacheFuzzy/results_cache_disabled_(concurrency:_1) --- PASS: TestSplitAndCacheMiddleware_ResultsCacheFuzzy (14.84s) --- PASS: TestSplitAndCacheMiddleware_ResultsCacheFuzzy/split_by_interval_disabled_(concurrency:_10) (16.22s) --- PASS: TestSplitAndCacheMiddleware_ResultsCacheFuzzy/cache_unaligned_requests_(concurrency:_10) (12.49s) --- PASS: TestSplitAndCacheMiddleware_ResultsCacheFuzzy/cache_unaligned_requests_(concurrency:_1) (42.22s) --- PASS: TestSplitAndCacheMiddleware_ResultsCacheFuzzy/increased_max_cache_freshness_(concurrency:_1) (44.29s) --- PASS: TestSplitAndCacheMiddleware_ResultsCacheFuzzy/reduced_query_parallelism_(concurrency:_1) (45.97s) --- PASS: TestSplitAndCacheMiddleware_ResultsCacheFuzzy/increased_max_cache_freshness_(concurrency:_10) (26.51s) --- PASS: TestSplitAndCacheMiddleware_ResultsCacheFuzzy/reduced_query_parallelism_(concurrency:_10) (27.80s) --- PASS: TestSplitAndCacheMiddleware_ResultsCacheFuzzy/results_cache_disabled_(concurrency:_10) (26.83s) --- PASS: TestSplitAndCacheMiddleware_ResultsCacheFuzzy/default_config_(concurrency:_10) (14.02s) --- PASS: TestSplitAndCacheMiddleware_ResultsCacheFuzzy/both_split_by_interval_and_results_cache_disabled_(concurrency:_10) (15.23s) --- PASS: TestSplitAndCacheMiddleware_ResultsCacheFuzzy/split_by_interval_disabled_(concurrency:_1) (72.31s) --- PASS: TestSplitAndCacheMiddleware_ResultsCacheFuzzy/default_config_(concurrency:_1) (33.36s) --- PASS: TestSplitAndCacheMiddleware_ResultsCacheFuzzy/results_cache_disabled_(concurrency:_1) (23.01s) --- PASS: TestSplitAndCacheMiddleware_ResultsCacheFuzzy/both_split_by_interval_and_results_cache_disabled_(concurrency:_1) (30.70s) === RUN TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases === RUN TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_drop_tiny_extent_that_overlaps_with_non-tiny_request_only === RUN TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_replace_tiny_extents_that_are_cover_by_bigger_request === RUN TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_not_drop_tiny_extent_that_completely_overlaps_with_tiny_request === RUN TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_merge_fragmented_extents_if_request_fills_the_hole === RUN TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_left-extend_extent_if_request_starts_earlier_than_extent_in_cache === RUN TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_right-extend_extent_if_request_ends_later_than_extent_in_cache === RUN TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_not_drop_tiny_extent_that_partially_center-overlaps_with_tiny_request === RUN TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_not_drop_tiny_extent_that_partially_left-overlaps_with_tiny_request === RUN TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_not_drop_tiny_extent_that_partially_right-overlaps_with_tiny_request === RUN TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_not_throw_error_if_complete-overlapped_smaller_Extent_is_erroneous --- PASS: TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases (0.02s) --- PASS: TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_drop_tiny_extent_that_overlaps_with_non-tiny_request_only (0.00s) --- PASS: TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_replace_tiny_extents_that_are_cover_by_bigger_request (0.00s) --- PASS: TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_not_drop_tiny_extent_that_completely_overlaps_with_tiny_request (0.00s) --- PASS: TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_merge_fragmented_extents_if_request_fills_the_hole (0.00s) --- PASS: TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_left-extend_extent_if_request_starts_earlier_than_extent_in_cache (0.00s) --- PASS: TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_right-extend_extent_if_request_ends_later_than_extent_in_cache (0.00s) --- PASS: TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_not_drop_tiny_extent_that_partially_center-overlaps_with_tiny_request (0.00s) --- PASS: TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_not_drop_tiny_extent_that_partially_left-overlaps_with_tiny_request (0.00s) --- PASS: TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_not_drop_tiny_extent_that_partially_right-overlaps_with_tiny_request (0.00s) --- PASS: TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_not_throw_error_if_complete-overlapped_smaller_Extent_is_erroneous (0.00s) === RUN TestSplitAndCacheMiddleware_StoreAndFetchCacheExtents === RUN TestSplitAndCacheMiddleware_StoreAndFetchCacheExtents/fetchCacheExtents()_should_return_a_slice_with_the_same_number_of_input_keys_but_empty_extents_on_cache_miss === RUN TestSplitAndCacheMiddleware_StoreAndFetchCacheExtents/fetchCacheExtents()_should_return_a_slice_with_the_same_number_of_input_keys_and_some_extends_filled_up_on_partial_cache_hit === RUN TestSplitAndCacheMiddleware_StoreAndFetchCacheExtents/fetchCacheExtents()_should_not_return_an_extent_if_its_key_doesn't_match_the_requested_one_(hash_collision) --- PASS: TestSplitAndCacheMiddleware_StoreAndFetchCacheExtents (0.00s) --- PASS: TestSplitAndCacheMiddleware_StoreAndFetchCacheExtents/fetchCacheExtents()_should_return_a_slice_with_the_same_number_of_input_keys_but_empty_extents_on_cache_miss (0.00s) --- PASS: TestSplitAndCacheMiddleware_StoreAndFetchCacheExtents/fetchCacheExtents()_should_return_a_slice_with_the_same_number_of_input_keys_and_some_extends_filled_up_on_partial_cache_hit (0.00s) --- PASS: TestSplitAndCacheMiddleware_StoreAndFetchCacheExtents/fetchCacheExtents()_should_not_return_an_extent_if_its_key_doesn't_match_the_requested_one_(hash_collision) (0.00s) === RUN TestSplitAndCacheMiddleware_WrapMultipleTimes --- PASS: TestSplitAndCacheMiddleware_WrapMultipleTimes (0.00s) === RUN TestSplitRequests_prepareDownstreamRequests === RUN TestSplitRequests_prepareDownstreamRequests/should_return_an_empty_slice_on_no_downstream_requests === RUN TestSplitRequests_prepareDownstreamRequests/should_inject_ID_and_hints_on_downstream_requests_and_return_them --- PASS: TestSplitRequests_prepareDownstreamRequests (0.00s) --- PASS: TestSplitRequests_prepareDownstreamRequests/should_return_an_empty_slice_on_no_downstream_requests (0.00s) --- PASS: TestSplitRequests_prepareDownstreamRequests/should_inject_ID_and_hints_on_downstream_requests_and_return_them (0.00s) === RUN TestSplitRequests_storeDownstreamResponses === RUN TestSplitRequests_storeDownstreamResponses/should_associate_downstream_responses_to_requests === RUN TestSplitRequests_storeDownstreamResponses/should_return_error_if_a_downstream_response_is_missing === RUN TestSplitRequests_storeDownstreamResponses/should_return_error_if_multiple_downstream_responses_have_the_same_ID === RUN TestSplitRequests_storeDownstreamResponses/should_return_error_if_extra_downst