==> Building on reshiram ==> Attempt to run pkgver()... ==> Making package: mimir 2.5.0-1 (Tue 17 Jan 2023 07:50:02 PM EET) ==> Checking runtime dependencies... ==> Checking buildtime dependencies... ==> Retrieving sources...  -> Updating mimir git repo...  -> Found systemd.service  -> Found sysusers.conf  -> Found tmpfiles.conf  -> Found config.yaml ==> Validating source files with b2sums... ==> Extracting sources...  -> Creating working copy of mimir git repo... ==> Starting prepare()... ==> Starting pkgver()... ==> Sources are ready. ==> Checking for remote environment... ==> Syncing package to remote host... sending incremental file list ./ PKGBUILD 1,399 58% 0.00kB/s 0:00:00 2,381 100% 958.98kB/s 0:00:00 (xfr#1, to-chk=5/7) config.yaml 38 100% 37.11kB/s 0:00:00 38 100% 37.11kB/s 0:00:00 (xfr#2, to-chk=4/7) mimir-2.5.0-1.log 1,301 100% 1.24MB/s 0:00:00 1,301 100% 1.24MB/s 0:00:00 (xfr#3, to-chk=3/7) systemd.service 700 69% 683.59kB/s 0:00:00 1,011 100% 987.30kB/s 0:00:00 (xfr#4, to-chk=2/7) sysusers.conf 45 100% 43.95kB/s 0:00:00 45 100% 43.95kB/s 0:00:00 (xfr#5, to-chk=1/7) tmpfiles.conf 34 100% 33.20kB/s 0:00:00 34 100% 33.20kB/s 0:00:00 (xfr#6, to-chk=0/7) sent 1,380 bytes received 199 bytes 3,158.00 bytes/sec total size is 4,564 speedup is 2.89 ==> Patching arch to riscv64... ==> Running extra-riscv64-build -- -d /home/felix/packages/riscv64-pkg-cache:/var/cache/pacman/pkg -l felix3 on remote host... [?25l:: Synchronizing package databases... core downloading... extra downloading... community downloading... :: Starting full system upgrade... resolving dependencies... looking for conflicting packages... Package (1) Old Version New Version Net Change Download Size core/pacman 6.0.2-5.1 6.0.2-5.2 0.00 MiB 0.89 MiB Total Download Size: 0.89 MiB Total Installed Size: 4.62 MiB Net Upgrade Size: 0.00 MiB :: Proceed with installation? [Y/n] :: Retrieving packages... pacman-6.0.2-5.2-riscv64 downloading... checking keyring... checking package integrity... loading package files... checking for file conflicts... :: Processing package changes... upgrading pacman... warning: /etc/makepkg.conf installed as /etc/makepkg.conf.pacnew [?25h==> Building in chroot for [extra] (riscv64)... ==> Synchronizing chroot copy [/var/lib/archbuild/extra-riscv64/root] -> [felix3]...done ==> Making package: mimir 2.5.0-1 (Wed Jan 18 01:51:04 2023) ==> Retrieving sources...  -> Updating mimir git repo... From https://github.com/grafana/mimir - [deleted] (none) -> aasp/cve-2022-41717 - [deleted] (none) -> add-err-check-linter - [deleted] (none) -> charleskorn/mimirtool-homebrew-installation - [deleted] (none) -> charleskorn/mmap-less-index-header-reader-optimisation - [deleted] (none) -> charleskorn/mmap-less-index-header-reader-optimisation-single-pass - [deleted] (none) -> chore/alertmanager-tests-cleanup - [deleted] (none) -> dimitar/streaming-series-caching-more-efficient-protobuf - [deleted] (none) -> felix/alert-on-components-idle - [deleted] (none) -> leizor/sparsehistogram - [deleted] (none) -> osg-grafana/3275-1-restructure-helm-chart-documentation - [deleted] (none) -> poc-memcache-client-memory-pooling - [deleted] (none) -> prometheus-0db77b4c7 - [deleted] (none) -> remove-block-hash-support - [deleted] (none) -> refs/pull/2702/merge - [deleted] (none) -> refs/pull/3439/merge - [deleted] (none) -> refs/pull/3598/merge - [deleted] (none) -> refs/pull/3653/merge - [deleted] (none) -> refs/pull/3654/merge - [deleted] (none) -> refs/pull/3673/merge - [deleted] (none) -> refs/pull/3682/merge - [deleted] (none) -> refs/pull/3694/merge - [deleted] (none) -> refs/pull/3719/merge - [deleted] (none) -> refs/pull/3723/merge - [deleted] (none) -> refs/pull/3729/merge - [deleted] (none) -> refs/pull/3733/merge - [deleted] (none) -> refs/pull/3737/merge - [deleted] (none) -> refs/pull/3742/merge - [deleted] (none) -> refs/pull/3748/merge - [deleted] (none) -> refs/pull/3751/merge - [deleted] (none) -> refs/pull/3755/merge - [deleted] (none) -> refs/pull/3756/merge - [deleted] (none) -> refs/pull/3758/merge remote: Enumerating objects: 9430, done. remote: Counting objects: 0% (1/5993) remote: Counting objects: 1% (60/5993) remote: Counting objects: 2% (120/5993) remote: Counting objects: 3% (180/5993) remote: Counting objects: 4% (240/5993) remote: Counting objects: 5% (300/5993) remote: Counting objects: 6% (360/5993) remote: Counting objects: 7% (420/5993) remote: Counting objects: 8% (480/5993) remote: Counting objects: 9% (540/5993) remote: Counting objects: 10% (600/5993) remote: Counting objects: 11% (660/5993) remote: Counting objects: 12% (720/5993) remote: Counting objects: 13% (780/5993) remote: Counting objects: 14% (840/5993) remote: Counting objects: 15% (899/5993) remote: Counting objects: 16% (959/5993) remote: Counting objects: 17% (1019/5993) remote: Counting objects: 18% (1079/5993) remote: Counting objects: 19% (1139/5993) remote: Counting objects: 20% (1199/5993) remote: Counting objects: 21% (1259/5993) remote: Counting objects: 22% (1319/5993) remote: Counting objects: 23% (1379/5993) remote: Counting objects: 24% (1439/5993) remote: Counting objects: 25% (1499/5993) remote: Counting objects: 26% (1559/5993) remote: Counting objects: 27% (1619/5993) remote: Counting objects: 28% (1679/5993) remote: Counting objects: 29% (1738/5993) remote: Counting objects: 30% (1798/5993) remote: Counting objects: 31% (1858/5993) remote: Counting objects: 32% (1918/5993) remote: Counting objects: 33% (1978/5993) remote: Counting objects: 34% (2038/5993) remote: Counting objects: 35% (2098/5993) remote: Counting objects: 36% (2158/5993) remote: Counting objects: 37% (2218/5993) remote: Counting objects: 38% (2278/5993) remote: Counting objects: 39% (2338/5993) remote: Counting objects: 40% (2398/5993) remote: Counting objects: 41% (2458/5993) remote: Counting objects: 42% (2518/5993) remote: Counting objects: 43% (2577/5993) remote: Counting objects: 44% (2637/5993) remote: Counting objects: 45% (2697/5993) remote: Counting objects: 46% (2757/5993) remote: Counting objects: 47% (2817/5993) remote: Counting objects: 48% (2877/5993) remote: Counting objects: 49% (2937/5993) remote: Counting objects: 50% (2997/5993) remote: Counting objects: 51% (3057/5993) remote: Counting objects: 52% (3117/5993) remote: Counting objects: 53% (3177/5993) remote: Counting objects: 54% (3237/5993) remote: Counting objects: 55% (3297/5993) remote: Counting objects: 56% (3357/5993) remote: Counting objects: 57% (3417/5993) remote: Counting objects: 58% (3476/5993) remote: Counting objects: 59% (3536/5993) remote: Counting objects: 60% (3596/5993) remote: Counting objects: 61% (3656/5993) remote: Counting objects: 62% (3716/5993) remote: Counting objects: 63% (3776/5993) remote: Counting objects: 64% (3836/5993) remote: Counting objects: 65% (3896/5993) remote: Counting objects: 66% (3956/5993) remote: Counting objects: 67% (4016/5993) remote: Counting objects: 68% (4076/5993) remote: Counting objects: 69% (4136/5993) remote: Counting objects: 70% (4196/5993) remote: Counting objects: 71% (4256/5993) remote: Counting objects: 72% (4315/5993) remote: Counting objects: 73% (4375/5993) remote: Counting objects: 74% (4435/5993) remote: Counting objects: 75% (4495/5993) remote: Counting objects: 76% (4555/5993) remote: Counting objects: 77% (4615/5993) remote: Counting objects: 78% (4675/5993) remote: Counting objects: 79% (4735/5993) remote: Counting objects: 80% (4795/5993) remote: Counting objects: 81% (4855/5993) remote: Counting objects: 82% (4915/5993) remote: Counting objects: 83% (4975/5993) remote: Counting objects: 84% (5035/5993) remote: Counting objects: 85% (5095/5993) remote: Counting objects: 86% (5154/5993) remote: Counting objects: 87% (5214/5993) remote: Counting objects: 88% (5274/5993) remote: Counting objects: 89% (5334/5993) remote: Counting objects: 90% (5394/5993) remote: Counting objects: 91% (5454/5993) remote: Counting objects: 92% (5514/5993) remote: Counting objects: 93% (5574/5993) remote: Counting objects: 94% (5634/5993) remote: Counting objects: 95% (5694/5993) remote: Counting objects: 96% (5754/5993) remote: Counting objects: 97% (5814/5993) remote: Counting objects: 98% (5874/5993) remote: Counting objects: 99% (5934/5993) remote: Counting objects: 100% (5993/5993) remote: Counting objects: 100% (5993/5993), done. remote: Compressing objects: 0% (1/601) remote: Compressing objects: 1% (7/601) remote: Compressing objects: 2% (13/601) remote: Compressing objects: 3% (19/601) remote: Compressing objects: 4% (25/601) remote: Compressing objects: 5% (31/601) remote: Compressing objects: 6% (37/601) remote: Compressing objects: 7% (43/601) remote: Compressing objects: 8% (49/601) remote: Compressing objects: 9% (55/601) remote: Compressing objects: 10% (61/601) remote: Compressing objects: 11% (67/601) remote: Compressing objects: 12% (73/601) remote: Compressing objects: 13% (79/601) remote: Compressing objects: 14% (85/601) remote: Compressing objects: 15% (91/601) remote: Compressing objects: 16% (97/601) remote: Compressing objects: 17% (103/601) remote: Compressing objects: 18% (109/601) remote: Compressing objects: 19% (115/601) remote: Compressing objects: 20% (121/601) remote: Compressing objects: 21% (127/601) remote: Compressing objects: 22% (133/601) remote: Compressing objects: 23% (139/601) remote: Compressing objects: 24% (145/601) remote: Compressing objects: 25% (151/601) remote: Compressing objects: 26% (157/601) remote: Compressing objects: 27% (163/601) remote: Compressing objects: 28% (169/601) remote: Compressing objects: 29% (175/601) remote: Compressing objects: 30% (181/601) remote: Compressing objects: 31% (187/601) remote: Compressing objects: 32% (193/601) remote: Compressing objects: 33% (199/601) remote: Compressing objects: 34% (205/601) remote: Compressing objects: 35% (211/601) remote: Compressing objects: 36% (217/601) remote: Compressing objects: 37% (223/601) remote: Compressing objects: 38% (229/601) remote: Compressing objects: 39% (235/601) remote: Compressing objects: 40% (241/601) remote: Compressing objects: 41% (247/601) remote: Compressing objects: 42% (253/601) remote: Compressing objects: 43% (259/601) remote: Compressing objects: 44% (265/601) remote: Compressing objects: 45% (271/601) remote: Compressing objects: 46% (277/601) remote: Compressing objects: 47% (283/601) remote: Compressing objects: 48% (289/601) remote: Compressing objects: 49% (295/601) remote: Compressing objects: 50% (301/601) remote: Compressing objects: 51% (307/601) remote: Compressing objects: 52% (313/601) remote: Compressing objects: 53% (319/601) remote: Compressing objects: 54% (325/601) remote: Compressing objects: 55% (331/601) remote: Compressing objects: 56% (337/601) remote: Compressing objects: 57% (343/601) remote: Compressing objects: 58% (349/601) remote: Compressing objects: 59% (355/601) remote: Compressing objects: 60% (361/601) remote: Compressing objects: 61% (367/601) remote: Compressing objects: 62% (373/601) remote: Compressing objects: 63% (379/601) remote: Compressing objects: 64% (385/601) remote: Compressing objects: 65% (391/601) remote: Compressing objects: 66% (397/601) remote: Compressing objects: 67% (403/601) remote: Compressing objects: 68% (409/601) remote: Compressing objects: 69% (415/601) remote: Compressing objects: 70% (421/601) remote: Compressing objects: 71% (427/601) remote: Compressing objects: 72% (433/601) remote: Compressing objects: 73% (439/601) remote: Compressing objects: 74% (445/601) remote: Compressing objects: 75% (451/601) remote: Compressing objects: 76% (457/601) remote: Compressing objects: 77% (463/601) remote: Compressing objects: 78% (469/601) remote: Compressing objects: 79% (475/601) remote: Compressing objects: 80% (481/601) remote: Compressing objects: 81% (487/601) remote: Compressing objects: 82% (493/601) remote: Compressing objects: 83% (499/601) remote: Compressing objects: 84% (505/601) remote: Compressing objects: 85% (511/601) remote: Compressing objects: 86% (517/601) remote: Compressing objects: 87% (523/601) remote: Compressing objects: 88% (529/601) remote: Compressing objects: 89% (535/601) remote: Compressing objects: 90% (541/601) remote: Compressing objects: 91% (547/601) remote: Compressing objects: 92% (553/601) remote: Compressing objects: 93% (559/601) remote: Compressing objects: 94% (565/601) remote: Compressing objects: 95% (571/601) remote: Compressing objects: 96% (577/601) remote: Compressing objects: 97% (583/601) remote: Compressing objects: 98% (589/601) remote: Compressing objects: 99% (595/601) remote: Compressing objects: 100% (601/601) remote: Compressing objects: 100% (601/601), done. Receiving objects: 0% (1/9430) Receiving objects: 1% (95/9430) Receiving objects: 2% (189/9430) Receiving objects: 3% (283/9430) Receiving objects: 4% (378/9430) Receiving objects: 5% (472/9430) Receiving objects: 6% (566/9430) Receiving objects: 7% (661/9430) Receiving objects: 8% (755/9430) Receiving objects: 9% (849/9430) Receiving objects: 10% (943/9430) Receiving objects: 11% (1038/9430) Receiving objects: 12% (1132/9430) Receiving objects: 13% (1226/9430) Receiving objects: 14% (1321/9430) Receiving objects: 15% (1415/9430) Receiving objects: 16% (1509/9430) Receiving objects: 17% (1604/9430) Receiving objects: 18% (1698/9430) Receiving objects: 19% (1792/9430) Receiving objects: 20% (1886/9430) Receiving objects: 21% (1981/9430) Receiving objects: 22% (2075/9430) Receiving objects: 23% (2169/9430) Receiving objects: 24% (2264/9430) Receiving objects: 25% (2358/9430) Receiving objects: 26% (2452/9430) Receiving objects: 27% (2547/9430) Receiving objects: 28% (2641/9430) Receiving objects: 29% (2735/9430) Receiving objects: 30% (2829/9430) Receiving objects: 31% (2924/9430) Receiving objects: 32% (3018/9430) Receiving objects: 33% (3112/9430) Receiving objects: 34% (3207/9430) Receiving objects: 35% (3301/9430) Receiving objects: 36% (3395/9430) Receiving objects: 37% (3490/9430) Receiving objects: 38% (3584/9430) Receiving objects: 39% (3678/9430) Receiving objects: 40% (3772/9430) Receiving objects: 41% (3867/9430) Receiving objects: 42% (3961/9430) Receiving objects: 43% (4055/9430) Receiving objects: 44% (4150/9430) Receiving objects: 45% (4244/9430) Receiving objects: 46% (4338/9430) Receiving objects: 47% (4433/9430) Receiving objects: 48% (4527/9430) Receiving objects: 49% (4621/9430) Receiving objects: 50% (4715/9430) Receiving objects: 51% (4810/9430) Receiving objects: 52% (4904/9430) Receiving objects: 53% (4998/9430) Receiving objects: 54% (5093/9430) Receiving objects: 55% (5187/9430) Receiving objects: 56% (5281/9430) Receiving objects: 57% (5376/9430) Receiving objects: 58% (5470/9430) Receiving objects: 59% (5564/9430) Receiving objects: 60% (5658/9430) Receiving objects: 61% (5753/9430) Receiving objects: 62% (5847/9430) Receiving objects: 63% (5941/9430) Receiving objects: 64% (6036/9430) Receiving objects: 65% (6130/9430) Receiving objects: 66% (6224/9430) Receiving objects: 67% (6319/9430) Receiving objects: 68% (6413/9430) Receiving objects: 69% (6507/9430) Receiving objects: 70% (6601/9430) Receiving objects: 71% (6696/9430), 1.56 MiB | 3.11 MiB/s Receiving objects: 72% (6790/9430), 1.56 MiB | 3.11 MiB/s Receiving objects: 73% (6884/9430), 1.56 MiB | 3.11 MiB/s Receiving objects: 74% (6979/9430), 1.56 MiB | 3.11 MiB/s Receiving objects: 75% (7073/9430), 1.56 MiB | 3.11 MiB/s Receiving objects: 76% (7167/9430), 1.56 MiB | 3.11 MiB/s Receiving objects: 77% (7262/9430), 1.56 MiB | 3.11 MiB/s Receiving objects: 78% (7356/9430), 1.56 MiB | 3.11 MiB/s Receiving objects: 79% (7450/9430), 1.56 MiB | 3.11 MiB/s Receiving objects: 80% (7544/9430), 1.56 MiB | 3.11 MiB/s Receiving objects: 81% (7639/9430), 1.56 MiB | 3.11 MiB/s Receiving objects: 82% (7733/9430), 1.56 MiB | 3.11 MiB/s Receiving objects: 83% (7827/9430), 1.56 MiB | 3.11 MiB/s Receiving objects: 84% (7922/9430), 1.56 MiB | 3.11 MiB/s Receiving objects: 85% (8016/9430), 1.56 MiB | 3.11 MiB/s Receiving objects: 86% (8110/9430), 1.56 MiB | 3.11 MiB/s Receiving objects: 87% (8205/9430), 1.56 MiB | 3.11 MiB/s Receiving objects: 88% (8299/9430), 1.56 MiB | 3.11 MiB/s Receiving objects: 89% (8393/9430), 1.56 MiB | 3.11 MiB/s Receiving objects: 90% (8487/9430), 1.56 MiB | 3.11 MiB/s Receiving objects: 91% (8582/9430), 1.56 MiB | 3.11 MiB/s Receiving objects: 92% (8676/9430), 1.56 MiB | 3.11 MiB/s Receiving objects: 93% (8770/9430), 1.56 MiB | 3.11 MiB/s Receiving objects: 94% (8865/9430), 1.56 MiB | 3.11 MiB/s Receiving objects: 95% (8959/9430), 1.56 MiB | 3.11 MiB/s Receiving objects: 96% (9053/9430), 1.56 MiB | 3.11 MiB/s remote: Total 9430 (delta 5545), reused 5742 (delta 5388), pack-reused 3437 Receiving objects: 96% (9130/9430), 4.31 MiB | 4.30 MiB/s Receiving objects: 97% (9148/9430), 4.31 MiB | 4.30 MiB/s Receiving objects: 98% (9242/9430), 4.31 MiB | 4.30 MiB/s Receiving objects: 99% (9336/9430), 4.31 MiB | 4.30 MiB/s Receiving objects: 100% (9430/9430), 4.31 MiB | 4.30 MiB/s Receiving objects: 100% (9430/9430), 4.35 MiB | 4.31 MiB/s, done. Resolving deltas: 0% (0/6988) Resolving deltas: 1% (70/6988) Resolving deltas: 2% (140/6988) Resolving deltas: 3% (210/6988) Resolving deltas: 4% (280/6988) Resolving deltas: 5% (350/6988) Resolving deltas: 6% (420/6988) Resolving deltas: 7% (490/6988) Resolving deltas: 8% (561/6988) Resolving deltas: 9% (630/6988) Resolving deltas: 10% (699/6988) Resolving deltas: 11% (769/6988) Resolving deltas: 12% (839/6988) Resolving deltas: 13% (909/6988) Resolving deltas: 14% (979/6988) Resolving deltas: 15% (1049/6988) Resolving deltas: 16% (1119/6988) Resolving deltas: 17% (1188/6988) Resolving deltas: 18% (1258/6988) Resolving deltas: 19% (1328/6988) Resolving deltas: 20% (1398/6988) Resolving deltas: 21% (1468/6988) Resolving deltas: 22% (1538/6988) Resolving deltas: 23% (1609/6988) Resolving deltas: 24% (1678/6988) Resolving deltas: 25% (1747/6988) Resolving deltas: 26% (1817/6988) Resolving deltas: 27% (1888/6988) Resolving deltas: 28% (1957/6988) Resolving deltas: 29% (2027/6988) Resolving deltas: 30% (2097/6988) Resolving deltas: 31% (2167/6988) Resolving deltas: 32% (2237/6988) Resolving deltas: 33% (2307/6988) Resolving deltas: 34% (2376/6988) Resolving deltas: 35% (2446/6988) Resolving deltas: 36% (2516/6988) Resolving deltas: 37% (2586/6988) Resolving deltas: 38% (2656/6988) Resolving deltas: 39% (2726/6988) Resolving deltas: 40% (2796/6988) Resolving deltas: 41% (2866/6988) Resolving deltas: 42% (2935/6988) Resolving deltas: 43% (3005/6988) Resolving deltas: 44% (3075/6988) Resolving deltas: 45% (3145/6988) Resolving deltas: 46% (3215/6988) Resolving deltas: 47% (3285/6988) Resolving deltas: 48% (3355/6988) Resolving deltas: 49% (3425/6988) Resolving deltas: 50% (3494/6988) Resolving deltas: 51% (3564/6988) Resolving deltas: 52% (3634/6988) Resolving deltas: 53% (3704/6988) Resolving deltas: 54% (3774/6988) Resolving deltas: 55% (3844/6988) Resolving deltas: 56% (3914/6988) Resolving deltas: 57% (3984/6988) Resolving deltas: 58% (4054/6988) Resolving deltas: 59% (4123/6988) Resolving deltas: 60% (4193/6988) Resolving deltas: 61% (4263/6988) Resolving deltas: 62% (4333/6988) Resolving deltas: 63% (4403/6988) Resolving deltas: 64% (4473/6988) Resolving deltas: 65% (4543/6988) Resolving deltas: 66% (4613/6988) Resolving deltas: 67% (4682/6988) Resolving deltas: 68% (4752/6988) Resolving deltas: 69% (4822/6988) Resolving deltas: 70% (4892/6988) Resolving deltas: 71% (4962/6988) Resolving deltas: 71% (4968/6988) Resolving deltas: 72% (5032/6988) Resolving deltas: 73% (5102/6988) Resolving deltas: 73% (5131/6988) Resolving deltas: 74% (5172/6988) Resolving deltas: 75% (5241/6988) Resolving deltas: 76% (5311/6988) Resolving deltas: 77% (5381/6988) Resolving deltas: 78% (5451/6988) Resolving deltas: 79% (5521/6988) Resolving deltas: 80% (5591/6988) Resolving deltas: 80% (5656/6988) Resolving deltas: 81% (5661/6988) Resolving deltas: 82% (5731/6988) Resolving deltas: 83% (5801/6988) Resolving deltas: 84% (5870/6988) Resolving deltas: 85% (5940/6988) Resolving deltas: 86% (6010/6988) Resolving deltas: 87% (6080/6988) Resolving deltas: 88% (6150/6988) Resolving deltas: 88% (6180/6988) Resolving deltas: 89% (6220/6988) Resolving deltas: 90% (6290/6988) Resolving deltas: 91% (6360/6988) Resolving deltas: 92% (6429/6988) Resolving deltas: 93% (6499/6988) Resolving deltas: 94% (6569/6988) Resolving deltas: 94% (6591/6988) Resolving deltas: 95% (6639/6988) Resolving deltas: 96% (6709/6988) Resolving deltas: 97% (6779/6988) Resolving deltas: 98% (6849/6988) Resolving deltas: 99% (6919/6988) Resolving deltas: 99% (6950/6988) Resolving deltas: 100% (6988/6988) Resolving deltas: 100% (6988/6988), completed with 947 local objects. * [new branch] 56quarters/2.6-removal -> 56quarters/2.6-removal * [new branch] 56quarters/slice-pointer -> 56quarters/slice-pointer * [new branch] aasp/minio-update -> aasp/minio-update + 6c7a3ebaf...336e165fb add-default-am-config -> add-default-am-config (forced update) * [new branch] add-hacky-tool-to-directly-query-ingester -> add-hacky-tool-to-directly-query-ingester * [new branch] aggregations-v2 -> aggregations-v2 * [new branch] backport-3853-to-r218 -> backport-3853-to-r218 * [new branch] bohandley/prom-from-scratch -> bohandley/prom-from-scratch * [new branch] charleskorn/mmap-less-index-header-reader-buffered-reader-with-seek -> charleskorn/mmap-less-index-header-reader-buffered-reader-with-seek * [new branch] charleskorn/mmap-less-index-header-reader-no-seeking -> charleskorn/mmap-less-index-header-reader-no-seeking * [new branch] charleskorn/rule-evaluation-tool -> charleskorn/rule-evaluation-tool * [new branch] dimitar/copy-migrate-from-cortex -> dimitar/copy-migrate-from-cortex * [new branch] dimitar/query-replay -> dimitar/query-replay * [new branch] dimitar/r217-with-to-postings-group-duration -> dimitar/r217-with-to-postings-group-duration * [new branch] dimitar/remove-jsonnet-content-from-migrate-from-cortex -> dimitar/remove-jsonnet-content-from-migrate-from-cortex * [new branch] dimitar/st-gw-shortcut-queries-with-0-series-labelvalues -> dimitar/st-gw-shortcut-queries-with-0-series-labelvalues * [new branch] dimitar/st-gw-shortcut-queries-with-0-series-rebased-r219 -> dimitar/st-gw-shortcut-queries-with-0-series-rebased-r219 * [new branch] dimitar/store-gateway-chunks-grouping -> dimitar/store-gateway-chunks-grouping * [new branch] dimitar/streaming-postings -> dimitar/streaming-postings * [new branch] dimitar/update-mimirtool-confgig-convert-descriptors -> dimitar/update-mimirtool-confgig-convert-descriptors * [new branch] helm-chart-weekly-4.1.0-weekly.218 -> helm-chart-weekly-4.1.0-weekly.218 + 659e3739f...7beb731d8 labels-as-string -> labels-as-string (forced update) + b4112d2db...ff47cf608 lamida/admin-cache -> lamida/admin-cache (forced update) * [new branch] leizor/mimir/issues/3767 -> leizor/mimir/issues/3767 * [new branch] leizor/tsdb-chunks -> leizor/tsdb-chunks * [new branch] leizor/tsdb-index-health -> leizor/tsdb-index-health * [new branch] leizor/tsdb-print-chunk -> leizor/tsdb-print-chunk f449a69f4..0a5e0e33a main -> main * [new branch] mark_series_ephemeral_in_distributor -> mark_series_ephemeral_in_distributor * [new branch] mixin-include-container-logs-link -> mixin-include-container-logs-link * [new branch] pr3973 -> pr3973 * [new branch] r217 -> r217 * [new branch] r218 -> r218 * [new branch] r219 -> r219 * [new branch] r220 -> r220 * [new branch] r221 -> r221 * [new branch] read-write-deployment/docs -> read-write-deployment/docs * [new branch] refactor/compactor -> refactor/compactor ffd3dbd2e..c586d4c21 release-2.2 -> release-2.2 c8be0e3a3..75576aa75 release-2.3 -> release-2.3 65c4cc3cb..c2eaf3b3a release-2.4 -> release-2.4 09c887f80..1852e09b0 release-2.5 -> release-2.5 * [new branch] release-helm-4.0.1 -> release-helm-4.0.1 * [new branch] snyk-monitor-workflow -> snyk-monitor-workflow 0a014cd6e..9a2e392f1 sparsehistogram -> sparsehistogram + 1891ead1b...f721c7b8f sparsehistogram-goes-boom -> sparsehistogram-goes-boom (forced update) * [new branch] sparsehistogram-test -> sparsehistogram-test 46c9ad9fc..cbb10d89f store-gateway-load-test -> store-gateway-load-test * [new branch] support-multiple-querier-scaled-objects -> support-multiple-querier-scaled-objects * [new branch] update-otel-col-to-latest-release -> update-otel-col-to-latest-release + f529c7b00...dfdf9dff2 refs/pull/3198/merge -> refs/pull/3198/merge (forced update) + e95408cdd...9e21e1b9d refs/pull/3295/merge -> refs/pull/3295/merge (forced update) + 7e07621a5...d024edf70 refs/pull/3411/merge -> refs/pull/3411/merge (forced update) + 21220502c...5797ba481 refs/pull/3439/head -> refs/pull/3439/head (forced update) 761edb31f..8586e04c0 refs/pull/3457/head -> refs/pull/3457/head + e407dc8a5...cec73e0fc refs/pull/3457/merge -> refs/pull/3457/merge (forced update) 0a014cd6e..9a2e392f1 refs/pull/3505/head -> refs/pull/3505/head + 7ca057698...aeb91ca69 refs/pull/3505/merge -> refs/pull/3505/merge (forced update) + 6c7a3ebaf...336e165fb refs/pull/3541/head -> refs/pull/3541/head (forced update) + 8904b213f...47ce85096 refs/pull/3541/merge -> refs/pull/3541/merge (forced update) + 5401fd9d2...525911bdb refs/pull/3554/merge -> refs/pull/3554/merge (forced update) + 659e3739f...7beb731d8 refs/pull/3555/head -> refs/pull/3555/head (forced update) + 5981968fd...c9056ae04 refs/pull/3555/merge -> refs/pull/3555/merge (forced update) eb520bb62..fb6935f27 refs/pull/3598/head -> refs/pull/3598/head + deb626639...435a56ebd refs/pull/3619/merge -> refs/pull/3619/merge (forced update) + 46b40948a...2224ef106 refs/pull/3629/merge -> refs/pull/3629/merge (forced update) + 22f626258...6fc329f4f refs/pull/3637/merge -> refs/pull/3637/merge (forced update) + 54ca80879...ab9903bbe refs/pull/3653/head -> refs/pull/3653/head (forced update) 4de050e4f..ebd766933 refs/pull/3682/head -> refs/pull/3682/head + 1891ead1b...f721c7b8f refs/pull/3707/head -> refs/pull/3707/head (forced update) + 53f9814e9...c38cd708b refs/pull/3707/merge -> refs/pull/3707/merge (forced update) + 2b5e16e01...58c0ac020 refs/pull/3719/head -> refs/pull/3719/head (forced update) + 230215d6a...2782f9a2d refs/pull/3723/head -> refs/pull/3723/head (forced update) + 950bdb145...2ebcac882 refs/pull/3729/head -> refs/pull/3729/head (forced update) + b4112d2db...ff47cf608 refs/pull/3740/head -> refs/pull/3740/head (forced update) + 8beeaeee5...b1223b092 refs/pull/3740/merge -> refs/pull/3740/merge (forced update) + 847e51e40...1066f9c80 refs/pull/3742/head -> refs/pull/3742/head (forced update) + f6e9ebf6f...99f86578d refs/pull/3751/head -> refs/pull/3751/head (forced update) + a9dc643fa...c78e5e523 refs/pull/3755/head -> refs/pull/3755/head (forced update) 4664f4a79..1d590c50c refs/pull/3756/head -> refs/pull/3756/head + de8f1a080...f38f4ef9d refs/pull/3758/head -> refs/pull/3758/head (forced update) * [new ref] refs/pull/3763/head -> refs/pull/3763/head * [new ref] refs/pull/3765/head -> refs/pull/3765/head * [new ref] refs/pull/3766/head -> refs/pull/3766/head * [new ref] refs/pull/3768/head -> refs/pull/3768/head * [new ref] refs/pull/3769/head -> refs/pull/3769/head * [new ref] refs/pull/3770/head -> refs/pull/3770/head * [new ref] refs/pull/3771/head -> refs/pull/3771/head * [new ref] refs/pull/3773/head -> refs/pull/3773/head * [new ref] refs/pull/3774/head -> refs/pull/3774/head * [new ref] refs/pull/3775/head -> refs/pull/3775/head * [new ref] refs/pull/3776/head -> refs/pull/3776/head * [new ref] refs/pull/3779/head -> refs/pull/3779/head * [new ref] refs/pull/3780/head -> refs/pull/3780/head * [new ref] refs/pull/3781/head -> refs/pull/3781/head * [new ref] refs/pull/3782/head -> refs/pull/3782/head * [new ref] refs/pull/3784/head -> refs/pull/3784/head * [new ref] refs/pull/3784/merge -> refs/pull/3784/merge * [new ref] refs/pull/3785/head -> refs/pull/3785/head * [new ref] refs/pull/3787/head -> refs/pull/3787/head * [new ref] refs/pull/3789/head -> refs/pull/3789/head * [new ref] refs/pull/3790/head -> refs/pull/3790/head * [new ref] refs/pull/3793/head -> refs/pull/3793/head * [new ref] refs/pull/3794/head -> refs/pull/3794/head * [new ref] refs/pull/3795/head -> refs/pull/3795/head * [new ref] refs/pull/3796/head -> refs/pull/3796/head * [new ref] refs/pull/3797/head -> refs/pull/3797/head * [new ref] refs/pull/3798/head -> refs/pull/3798/head * [new ref] refs/pull/3798/merge -> refs/pull/3798/merge * [new ref] refs/pull/3799/head -> refs/pull/3799/head * [new ref] refs/pull/3800/head -> refs/pull/3800/head * [new ref] refs/pull/3802/head -> refs/pull/3802/head * [new ref] refs/pull/3803/head -> refs/pull/3803/head * [new ref] refs/pull/3809/head -> refs/pull/3809/head * [new ref] refs/pull/3810/head -> refs/pull/3810/head * [new ref] refs/pull/3811/head -> refs/pull/3811/head * [new ref] refs/pull/3812/head -> refs/pull/3812/head * [new ref] refs/pull/3816/head -> refs/pull/3816/head * [new ref] refs/pull/3817/head -> refs/pull/3817/head * [new ref] refs/pull/3820/head -> refs/pull/3820/head * [new ref] refs/pull/3821/head -> refs/pull/3821/head * [new ref] refs/pull/3822/head -> refs/pull/3822/head * [new ref] refs/pull/3823/head -> refs/pull/3823/head * [new ref] refs/pull/3825/head -> refs/pull/3825/head * [new ref] refs/pull/3826/head -> refs/pull/3826/head * [new ref] refs/pull/3827/head -> refs/pull/3827/head * [new ref] refs/pull/3828/head -> refs/pull/3828/head * [new ref] refs/pull/3829/head -> refs/pull/3829/head * [new ref] refs/pull/3831/head -> refs/pull/3831/head * [new ref] refs/pull/3836/head -> refs/pull/3836/head * [new ref] refs/pull/3837/head -> refs/pull/3837/head * [new ref] refs/pull/3839/head -> refs/pull/3839/head * [new ref] refs/pull/3840/head -> refs/pull/3840/head * [new ref] refs/pull/3842/head -> refs/pull/3842/head * [new ref] refs/pull/3845/head -> refs/pull/3845/head * [new ref] refs/pull/3846/head -> refs/pull/3846/head * [new ref] refs/pull/3848/head -> refs/pull/3848/head * [new ref] refs/pull/3849/head -> refs/pull/3849/head * [new ref] refs/pull/3851/head -> refs/pull/3851/head * [new ref] refs/pull/3852/head -> refs/pull/3852/head * [new ref] refs/pull/3852/merge -> refs/pull/3852/merge * [new ref] refs/pull/3853/head -> refs/pull/3853/head * [new ref] refs/pull/3854/head -> refs/pull/3854/head * [new ref] refs/pull/3855/head -> refs/pull/3855/head * [new ref] refs/pull/3856/head -> refs/pull/3856/head * [new ref] refs/pull/3857/head -> refs/pull/3857/head * [new ref] refs/pull/3858/head -> refs/pull/3858/head * [new ref] refs/pull/3858/merge -> refs/pull/3858/merge * [new ref] refs/pull/3859/head -> refs/pull/3859/head * [new ref] refs/pull/3862/head -> refs/pull/3862/head * [new ref] refs/pull/3862/merge -> refs/pull/3862/merge * [new ref] refs/pull/3863/head -> refs/pull/3863/head * [new ref] refs/pull/3864/head -> refs/pull/3864/head * [new ref] refs/pull/3866/head -> refs/pull/3866/head * [new ref] refs/pull/3866/merge -> refs/pull/3866/merge * [new ref] refs/pull/3867/head -> refs/pull/3867/head * [new ref] refs/pull/3868/head -> refs/pull/3868/head * [new ref] refs/pull/3869/head -> refs/pull/3869/head * [new ref] refs/pull/3870/head -> refs/pull/3870/head * [new ref] refs/pull/3871/head -> refs/pull/3871/head * [new ref] refs/pull/3872/head -> refs/pull/3872/head * [new ref] refs/pull/3872/merge -> refs/pull/3872/merge * [new ref] refs/pull/3873/head -> refs/pull/3873/head * [new ref] refs/pull/3874/head -> refs/pull/3874/head * [new ref] refs/pull/3875/head -> refs/pull/3875/head * [new ref] refs/pull/3877/head -> refs/pull/3877/head * [new ref] refs/pull/3879/head -> refs/pull/3879/head * [new ref] refs/pull/3880/head -> refs/pull/3880/head * [new ref] refs/pull/3881/head -> refs/pull/3881/head * [new ref] refs/pull/3882/head -> refs/pull/3882/head * [new ref] refs/pull/3883/head -> refs/pull/3883/head * [new ref] refs/pull/3885/head -> refs/pull/3885/head * [new ref] refs/pull/3886/head -> refs/pull/3886/head * [new ref] refs/pull/3887/head -> refs/pull/3887/head * [new ref] refs/pull/3888/head -> refs/pull/3888/head * [new ref] refs/pull/3889/head -> refs/pull/3889/head * [new ref] refs/pull/3890/head -> refs/pull/3890/head * [new ref] refs/pull/3891/head -> refs/pull/3891/head * [new ref] refs/pull/3892/head -> refs/pull/3892/head * [new ref] refs/pull/3893/head -> refs/pull/3893/head * [new ref] refs/pull/3894/head -> refs/pull/3894/head * [new ref] refs/pull/3895/head -> refs/pull/3895/head * [new ref] refs/pull/3896/head -> refs/pull/3896/head * [new ref] refs/pull/3897/head -> refs/pull/3897/head * [new ref] refs/pull/3897/merge -> refs/pull/3897/merge * [new ref] refs/pull/3898/head -> refs/pull/3898/head * [new ref] refs/pull/3899/head -> refs/pull/3899/head * [new ref] refs/pull/3901/head -> refs/pull/3901/head * [new ref] refs/pull/3902/head -> refs/pull/3902/head * [new ref] refs/pull/3905/head -> refs/pull/3905/head * [new ref] refs/pull/3906/head -> refs/pull/3906/head * [new ref] refs/pull/3907/head -> refs/pull/3907/head * [new ref] refs/pull/3908/head -> refs/pull/3908/head * [new ref] refs/pull/3910/head -> refs/pull/3910/head * [new ref] refs/pull/3911/head -> refs/pull/3911/head * [new ref] refs/pull/3911/merge -> refs/pull/3911/merge * [new ref] refs/pull/3912/head -> refs/pull/3912/head * [new ref] refs/pull/3913/head -> refs/pull/3913/head * [new ref] refs/pull/3914/head -> refs/pull/3914/head * [new ref] refs/pull/3915/head -> refs/pull/3915/head * [new ref] refs/pull/3916/head -> refs/pull/3916/head * [new ref] refs/pull/3917/head -> refs/pull/3917/head * [new ref] refs/pull/3918/head -> refs/pull/3918/head * [new ref] refs/pull/3919/head -> refs/pull/3919/head * [new ref] refs/pull/3920/head -> refs/pull/3920/head * [new ref] refs/pull/3921/head -> refs/pull/3921/head * [new ref] refs/pull/3922/head -> refs/pull/3922/head * [new ref] refs/pull/3923/head -> refs/pull/3923/head * [new ref] refs/pull/3927/head -> refs/pull/3927/head * [new ref] refs/pull/3928/head -> refs/pull/3928/head * [new ref] refs/pull/3929/head -> refs/pull/3929/head * [new ref] refs/pull/3929/merge -> refs/pull/3929/merge * [new ref] refs/pull/3930/head -> refs/pull/3930/head * [new ref] refs/pull/3930/merge -> refs/pull/3930/merge * [new ref] refs/pull/3932/head -> refs/pull/3932/head * [new ref] refs/pull/3934/head -> refs/pull/3934/head * [new ref] refs/pull/3936/head -> refs/pull/3936/head * [new ref] refs/pull/3937/head -> refs/pull/3937/head * [new ref] refs/pull/3938/head -> refs/pull/3938/head * [new ref] refs/pull/3942/head -> refs/pull/3942/head * [new ref] refs/pull/3942/merge -> refs/pull/3942/merge * [new ref] refs/pull/3943/head -> refs/pull/3943/head * [new ref] refs/pull/3946/head -> refs/pull/3946/head * [new ref] refs/pull/3946/merge -> refs/pull/3946/merge * [new ref] refs/pull/3948/head -> refs/pull/3948/head * [new ref] refs/pull/3948/merge -> refs/pull/3948/merge * [new ref] refs/pull/3950/head -> refs/pull/3950/head * [new ref] refs/pull/3950/merge -> refs/pull/3950/merge * [new ref] refs/pull/3951/head -> refs/pull/3951/head * [new ref] refs/pull/3951/merge -> refs/pull/3951/merge * [new ref] refs/pull/3952/head -> refs/pull/3952/head * [new ref] refs/pull/3952/merge -> refs/pull/3952/merge * [new ref] refs/pull/3953/head -> refs/pull/3953/head * [new ref] refs/pull/3954/head -> refs/pull/3954/head * [new ref] refs/pull/3955/head -> refs/pull/3955/head * [new ref] refs/pull/3956/head -> refs/pull/3956/head * [new ref] refs/pull/3956/merge -> refs/pull/3956/merge * [new ref] refs/pull/3957/head -> refs/pull/3957/head * [new ref] refs/pull/3957/merge -> refs/pull/3957/merge * [new ref] refs/pull/3959/head -> refs/pull/3959/head * [new ref] refs/pull/3960/head -> refs/pull/3960/head * [new ref] refs/pull/3961/head -> refs/pull/3961/head * [new ref] refs/pull/3962/head -> refs/pull/3962/head * [new ref] refs/pull/3962/merge -> refs/pull/3962/merge * [new ref] refs/pull/3963/head -> refs/pull/3963/head * [new ref] refs/pull/3964/head -> refs/pull/3964/head * [new ref] refs/pull/3964/merge -> refs/pull/3964/merge * [new ref] refs/pull/3965/head -> refs/pull/3965/head * [new ref] refs/pull/3966/head -> refs/pull/3966/head * [new ref] refs/pull/3967/head -> refs/pull/3967/head * [new ref] refs/pull/3968/head -> refs/pull/3968/head * [new ref] refs/pull/3968/merge -> refs/pull/3968/merge * [new ref] refs/pull/3972/head -> refs/pull/3972/head * [new ref] refs/pull/3973/head -> refs/pull/3973/head * [new ref] refs/pull/3974/head -> refs/pull/3974/head * [new ref] refs/pull/3975/head -> refs/pull/3975/head * [new ref] refs/pull/3976/head -> refs/pull/3976/head * [new ref] refs/pull/3978/head -> refs/pull/3978/head * [new ref] refs/pull/3978/merge -> refs/pull/3978/merge * [new ref] refs/pull/3982/head -> refs/pull/3982/head * [new ref] refs/pull/3982/merge -> refs/pull/3982/merge * [new tag] mimir-distributed-4.0.1 -> mimir-distributed-4.0.1 * [new tag] mimir-distributed-4.1.0-weekly.217 -> mimir-distributed-4.1.0-weekly.217 * [new tag] mimir-distributed-4.1.0-weekly.219 -> mimir-distributed-4.1.0-weekly.219 * [new tag] mimir-distributed-4.1.0-weekly.220 -> mimir-distributed-4.1.0-weekly.220  -> Found systemd.service  -> Found sysusers.conf  -> Found tmpfiles.conf  -> Found config.yaml ==> Validating source files with b2sums... mimir ... Skipped systemd.service ... Passed sysusers.conf ... Passed tmpfiles.conf ... Passed config.yaml ... Passed ==> Making package: mimir 2.5.0-1 (Wed Jan 18 01:52:02 2023) ==> Checking runtime dependencies... ==> Checking buildtime dependencies... ==> Installing missing dependencies... [?25lresolving dependencies... looking for conflicting packages... Package (5) New Version Net Change extra/perl-error 0.17029-4 0.04 MiB extra/perl-mailtools 2.21-6 0.11 MiB extra/perl-timedate 2.33-4 0.08 MiB extra/git 2.39.0-1 33.01 MiB community/go 2:1.19.3-1 434.45 MiB Total Installed Size: 467.69 MiB :: Proceed with installation? [Y/n] checking keyring... checking package integrity... loading package files... checking for file conflicts... :: Processing package changes... installing perl-error... installing perl-timedate... installing perl-mailtools... installing git... Optional dependencies for git tk: gitk and git gui perl-libwww: git svn perl-term-readkey: git svn and interactive.singlekey setting perl-io-socket-ssl: git send-email TLS support perl-authen-sasl: git send-email TLS support perl-mediawiki-api: git mediawiki support perl-datetime-format-iso8601: git mediawiki support perl-lwp-protocol-https: git mediawiki https support perl-cgi: gitweb (web interface) support python: git svn & git p4 subversion: git svn org.freedesktop.secrets: keyring credential helper libsecret: libsecret credential helper [installed] installing go... :: Running post-transaction hooks... (1/1) Warn about old perl modules [?25h==> Retrieving sources...  -> Found systemd.service  -> Found sysusers.conf  -> Found tmpfiles.conf  -> Found config.yaml ==> WARNING: Skipping all source file integrity checks. ==> Extracting sources...  -> Creating working copy of mimir git repo... Cloning into 'mimir'... done. Updating files: 18% (1612/8681) Updating files: 19% (1650/8681) Updating files: 20% (1737/8681) Updating files: 21% (1824/8681) Updating files: 22% (1910/8681) Updating files: 23% (1997/8681) Updating files: 24% (2084/8681) Updating files: 25% (2171/8681) Updating files: 26% (2258/8681) Updating files: 27% (2344/8681) Updating files: 28% (2431/8681) Updating files: 29% (2518/8681) Updating files: 30% (2605/8681) Updating files: 31% (2692/8681) Updating files: 31% (2720/8681) Updating files: 32% (2778/8681) Updating files: 33% (2865/8681) Updating files: 34% (2952/8681) Updating files: 35% (3039/8681) Updating files: 36% (3126/8681) Updating files: 37% (3212/8681) Updating files: 38% (3299/8681) Updating files: 39% (3386/8681) Updating files: 40% (3473/8681) Updating files: 41% (3560/8681) Updating files: 42% (3647/8681) Updating files: 43% (3733/8681) Updating files: 44% (3820/8681) Updating files: 45% (3907/8681) Updating files: 46% (3994/8681) Updating files: 47% (4081/8681) Updating files: 48% (4167/8681) Updating files: 49% (4254/8681) Updating files: 50% (4341/8681) Updating files: 50% (4375/8681) Updating files: 51% (4428/8681) Updating files: 52% (4515/8681) Updating files: 53% (4601/8681) Updating files: 54% (4688/8681) Updating files: 55% (4775/8681) Updating files: 56% (4862/8681) Updating files: 57% (4949/8681) Updating files: 58% (5035/8681) Updating files: 59% (5122/8681) Updating files: 60% (5209/8681) Updating files: 61% (5296/8681) Updating files: 62% (5383/8681) Updating files: 63% (5470/8681) Updating files: 64% (5556/8681) Updating files: 65% (5643/8681) Updating files: 66% (5730/8681) Updating files: 67% (5817/8681) Updating files: 68% (5904/8681) Updating files: 68% (5935/8681) Updating files: 69% (5990/8681) Updating files: 70% (6077/8681) Updating files: 71% (6164/8681) Updating files: 72% (6251/8681) Updating files: 73% (6338/8681) Updating files: 74% (6424/8681) Updating files: 75% (6511/8681) Updating files: 76% (6598/8681) Updating files: 77% (6685/8681) Updating files: 78% (6772/8681) Updating files: 79% (6858/8681) Updating files: 80% (6945/8681) Updating files: 81% (7032/8681) Updating files: 82% (7119/8681) Updating files: 83% (7206/8681) Updating files: 84% (7293/8681) Updating files: 85% (7379/8681) Updating files: 86% (7466/8681) Updating files: 87% (7553/8681) Updating files: 88% (7640/8681) Updating files: 88% (7643/8681) Updating files: 89% (7727/8681) Updating files: 90% (7813/8681) Updating files: 91% (7900/8681) Updating files: 92% (7987/8681) Updating files: 93% (8074/8681) Updating files: 94% (8161/8681) Updating files: 95% (8247/8681) Updating files: 96% (8334/8681) Updating files: 97% (8421/8681) Updating files: 98% (8508/8681) Updating files: 99% (8595/8681) Updating files: 99% (8650/8681) Updating files: 100% (8681/8681) Updating files: 100% (8681/8681), done. Updating files: 71% (1347/1871) Updating files: 72% (1348/1871) Updating files: 73% (1366/1871) Updating files: 74% (1385/1871) Updating files: 75% (1404/1871) Updating files: 76% (1422/1871) Updating files: 77% (1441/1871) Updating files: 78% (1460/1871) Updating files: 79% (1479/1871) Updating files: 80% (1497/1871) Updating files: 81% (1516/1871) Updating files: 82% (1535/1871) Updating files: 83% (1553/1871) Updating files: 84% (1572/1871) Updating files: 85% (1591/1871) Updating files: 86% (1610/1871) Updating files: 87% (1628/1871) Updating files: 88% (1647/1871) Updating files: 89% (1666/1871) Updating files: 90% (1684/1871) Updating files: 91% (1703/1871) Updating files: 92% (1722/1871) Updating files: 93% (1741/1871) Updating files: 94% (1759/1871) Updating files: 95% (1778/1871) Updating files: 96% (1797/1871) Updating files: 97% (1815/1871) Updating files: 98% (1834/1871) Updating files: 99% (1853/1871) Updating files: 100% (1871/1871) Updating files: 100% (1871/1871), done. Switched to a new branch 'makepkg' ==> Starting prepare()... ==> Starting pkgver()... ==> Starting build()... all modules verified internal/unsafeheader internal/goexperiment internal/goarch internal/cpu internal/goos runtime/internal/atomic internal/abi runtime/internal/math runtime/internal/sys runtime/internal/syscall internal/bytealg internal/race sync/atomic unicode unicode/utf8 internal/itoa encoding math/bits runtime unicode/utf16 container/list math crypto/internal/subtle crypto/subtle crypto/internal/boring/sig vendor/golang.org/x/crypto/cryptobyte/asn1 internal/nettrace vendor/golang.org/x/crypto/internal/subtle google.golang.org/protobuf/internal/flags google.golang.org/protobuf/internal/set github.com/Azure/azure-sdk-for-go/sdk/azcore/cloud github.com/Azure/azure-sdk-for-go/sdk/internal/errorinfo github.com/AzureAD/microsoft-authentication-library-for-go/apps/cache github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops/internal/grant github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/version github.com/Azure/azure-sdk-for-go/sdk/azcore/to github.com/googleapis/gax-go/v2/internal google.golang.org/grpc/attributes google.golang.org/grpc/serviceconfig github.com/golang/groupcache/lru go.opencensus.io go.opencensus.io/trace/internal golang.org/x/xerrors/internal cloud.google.com/go/storage/internal go.opencensus.io/internal/tagencoding github.com/google/go-cmp/cmp/internal/flags github.com/weaveworks/common/errors github.com/uber/jaeger-client-go/internal/baggage github.com/uber/jaeger-client-go/internal/reporterstats github.com/uber/jaeger-client-go/internal/throttler github.com/aws/aws-sdk-go-v2/internal/sdkio github.com/aws/aws-sdk-go-v2 golang.org/x/net/internal/iana github.com/prometheus/prometheus/model/value github.com/dennwc/varint golang.org/x/exp/constraints github.com/aws/aws-sdk-go/aws/client/metadata golang.org/x/exp/slices github.com/aws/aws-sdk-go/internal/sdkmath github.com/prometheus/prometheus/tsdb/goversion go.opentelemetry.io/otel/internal github.com/grafana/dskit/internal/math github.com/grafana/dskit/ring/util go.mongodb.org/mongo-driver/bson/bsonoptions go.mongodb.org/mongo-driver/bson/bsontype go.opentelemetry.io/collector/semconv/v1.6.1 go.opentelemetry.io/otel/metric/unit github.com/segmentio/fasthash/fnv1a go.opentelemetry.io/otel/metric/instrument image/color runtime/metrics internal/reflectlite sync internal/testlog internal/singleflight math/rand google.golang.org/protobuf/internal/pragma runtime/cgo internal/sysinfo google.golang.org/grpc/internal/buffer errors sort google.golang.org/grpc/internal/grpcsync github.com/aws/aws-sdk-go-v2/internal/sync/singleflight io internal/oserror syscall strconv bytes path reflect strings bufio time regexp/syntax internal/syscall/unix internal/syscall/execenv hash hash/crc32 crypto crypto/internal/randutil crypto/internal/nistec/fiat regexp context internal/poll internal/fmtsort io/fs encoding/binary crypto/rc4 os vendor/golang.org/x/net/dns/dnsmessage encoding/base64 crypto/cipher crypto/internal/nistec crypto/internal/boring crypto/des fmt crypto/aes crypto/sha512 crypto/internal/edwards25519/field crypto/hmac crypto/md5 crypto/internal/edwards25519 crypto/sha1 crypto/sha256 encoding/pem internal/godebug internal/intern flag encoding/json github.com/go-logfmt/logfmt log github.com/alecthomas/units github.com/pkg/errors compress/flate math/big encoding/hex net/netip compress/gzip github.com/go-kit/log net/url net github.com/go-kit/log/level path/filepath vendor/golang.org/x/crypto/chacha20 vendor/golang.org/x/crypto/internal/poly1305 vendor/golang.org/x/crypto/curve25519/internal/field vendor/golang.org/x/crypto/chacha20poly1305 vendor/golang.org/x/crypto/hkdf vendor/golang.org/x/text/transform crypto/rand crypto/elliptic crypto/internal/boring/bbig encoding/asn1 crypto/ed25519 crypto/rsa crypto/dsa vendor/golang.org/x/crypto/curve25519 vendor/golang.org/x/crypto/cryptobyte crypto/x509/pkix vendor/golang.org/x/text/unicode/bidi vendor/golang.org/x/text/unicode/norm crypto/ecdsa vendor/golang.org/x/net/http2/hpack vendor/golang.org/x/text/secure/bidirule mime mime/quotedprintable net/http/internal net/http/internal/ascii github.com/beorn7/perks/quantile github.com/cespare/xxhash/v2 hash/fnv go/token google.golang.org/protobuf/internal/detrand vendor/golang.org/x/net/idna google.golang.org/protobuf/internal/errors io/ioutil google.golang.org/protobuf/encoding/protowire google.golang.org/protobuf/internal/version github.com/prometheus/common/model google.golang.org/protobuf/reflect/protoreflect github.com/prometheus/common/internal/bitbucket.org/ww/goautoneg github.com/prometheus/procfs/internal/fs github.com/prometheus/procfs/internal/util golang.org/x/sys/unix runtime/debug google.golang.org/protobuf/internal/encoding/messageset google.golang.org/protobuf/internal/strs google.golang.org/protobuf/internal/genid google.golang.org/protobuf/internal/encoding/text google.golang.org/protobuf/internal/order google.golang.org/protobuf/reflect/protoregistry google.golang.org/protobuf/runtime/protoiface google.golang.org/protobuf/internal/encoding/defval google.golang.org/protobuf/proto google.golang.org/protobuf/internal/descfmt crypto/x509 net/textproto vendor/golang.org/x/net/http/httpguts vendor/golang.org/x/net/http/httpproxy mime/multipart google.golang.org/protobuf/encoding/prototext google.golang.org/protobuf/internal/descopts google.golang.org/protobuf/internal/filedesc github.com/prometheus/procfs github.com/efficientgo/tools/core/pkg/logerrcapture github.com/davecgh/go-spew/spew crypto/tls github.com/pmezard/go-difflib/difflib go.uber.org/goleak/internal/stack go.uber.org/goleak runtime/trace testing google.golang.org/protobuf/internal/encoding/tag google.golang.org/protobuf/internal/impl github.com/opentracing/opentracing-go/log github.com/efficientgo/tools/core/pkg/testutil golang.org/x/sync/errgroup golang.org/x/text/transform golang.org/x/text/unicode/bidi golang.org/x/text/unicode/norm net/http/httptrace golang.org/x/text/secure/bidirule encoding/xml net/http golang.org/x/net/idna github.com/Azure/azure-sdk-for-go/sdk/internal/log github.com/Azure/azure-sdk-for-go/sdk/azcore/log github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/log github.com/Azure/azure-sdk-for-go/sdk/internal/diag github.com/Azure/azure-sdk-for-go/sdk/internal/temporal github.com/Azure/azure-sdk-for-go/sdk/internal/uuid github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/json golang.org/x/net/http/httpguts github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/json/types/time github.com/kylelemons/godebug/diff github.com/kylelemons/godebug/pretty database/sql/driver github.com/google/uuid google.golang.org/protobuf/internal/filetype github.com/golang-jwt/jwt os/exec google.golang.org/protobuf/runtime/protoimpl google.golang.org/protobuf/types/descriptorpb google.golang.org/protobuf/types/known/timestamppb github.com/golang/protobuf/ptypes/timestamp github.com/pkg/browser golang.org/x/crypto/pkcs12/internal/rc2 gopkg.in/yaml.v2 golang.org/x/crypto/pkcs12 gopkg.in/yaml.v3 google.golang.org/protobuf/reflect/protodesc github.com/golang/protobuf/proto github.com/efficientgo/tools/core/pkg/merrors github.com/efficientgo/tools/core/pkg/errcapture google.golang.org/genproto/googleapis/rpc/code google.golang.org/protobuf/types/known/anypb github.com/prometheus/client_model/go github.com/matttproud/golang_protobuf_extensions/pbutil github.com/googleapis/gax-go/v2/apierror/internal/proto google.golang.org/api/internal/third_party/uritemplates expvar github.com/opentracing/opentracing-go github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared github.com/prometheus/client_golang/prometheus/internal github.com/prometheus/common/expfmt github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported github.com/thanos-io/objstore/tracing github.com/AzureAD/microsoft-authentication-library-for-go/apps/errors github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops/authority github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/shared github.com/Azure/azure-sdk-for-go/sdk/azcore/policy github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/pollers github.com/Azure/azure-sdk-for-go/sdk/azcore github.com/Azure/azure-sdk-for-go/sdk/azcore/streaming github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops/wstrust/defs github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/pollers/async github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/pollers/body github.com/prometheus/client_golang/prometheus github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/pollers/loc github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/pollers/op github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops/internal/comm github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops/wstrust github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops/accesstokens github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/local github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal github.com/thanos-io/objstore/exthttp github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops github.com/Azure/azure-sdk-for-go/sdk/storage/azblob github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth cloud.google.com/go/compute/metadata github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/base/internal/storage google.golang.org/api/googleapi google.golang.org/protobuf/types/known/durationpb github.com/prometheus/client_golang/prometheus/promauto github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/base google.golang.org/genproto/googleapis/rpc/errdetails github.com/grafana/dskit/flagext github.com/thanos-io/objstore github.com/AzureAD/microsoft-authentication-library-for-go/apps/confidential github.com/AzureAD/microsoft-authentication-library-for-go/apps/public google.golang.org/genproto/googleapis/rpc/status github.com/Azure/azure-sdk-for-go/sdk/azidentity google.golang.org/grpc/codes github.com/golang/protobuf/ptypes/any github.com/golang/protobuf/ptypes/duration github.com/thanos-io/objstore/providers/filesystem github.com/golang/protobuf/ptypes github.com/grafana/mimir/pkg/storage/bucket/filesystem google.golang.org/protobuf/internal/encoding/json google.golang.org/grpc/internal/status google.golang.org/grpc/status golang.org/x/net/internal/timeseries html google.golang.org/protobuf/encoding/protojson text/template/parse text/tabwriter google.golang.org/grpc/backoff google.golang.org/grpc/internal/grpclog google.golang.org/grpc/grpclog google.golang.org/grpc/connectivity github.com/googleapis/gax-go/v2/apierror google.golang.org/grpc/internal/credentials google.golang.org/grpc/credentials text/template google.golang.org/grpc/metadata google.golang.org/grpc/resolver google.golang.org/grpc/internal/grpcrand google.golang.org/grpc/credentials/insecure google.golang.org/grpc/internal google.golang.org/grpc/encoding google.golang.org/grpc/balancer google.golang.org/grpc/encoding/proto google.golang.org/grpc/internal/backoff google.golang.org/grpc/balancer/base google.golang.org/grpc/internal/balancerload google.golang.org/grpc/binarylog/grpc_binarylog_v1 google.golang.org/grpc/balancer/roundrobin google.golang.org/grpc/internal/grpcutil google.golang.org/grpc/internal/channelz google.golang.org/grpc/internal/binarylog html/template google.golang.org/grpc/internal/serviceconfig google.golang.org/grpc/balancer/grpclb/state google.golang.org/grpc/internal/resolver google.golang.org/grpc/internal/envconfig google.golang.org/grpc/internal/resolver/passthrough google.golang.org/grpc/internal/resolver/dns google.golang.org/grpc/internal/transport/networktype google.golang.org/grpc/internal/resolver/unix golang.org/x/net/http2/hpack google.golang.org/grpc/internal/metadata google.golang.org/grpc/internal/syscall golang.org/x/net/http2 golang.org/x/net/trace github.com/thanos-io/objstore/providers/azure google.golang.org/grpc/keepalive google.golang.org/grpc/peer google.golang.org/grpc/stats google.golang.org/grpc/tap net/http/httputil github.com/grafana/mimir/pkg/storage/bucket/azure google.golang.org/genproto/googleapis/api/annotations google.golang.org/genproto/googleapis/type/expr google.golang.org/protobuf/types/known/fieldmaskpb cloud.google.com/go/internal/optional go.opencensus.io/internal go.opencensus.io/trace/tracestate golang.org/x/xerrors cloud.google.com/go/internal/version google.golang.org/protobuf/types/known/emptypb go.opencensus.io/trace github.com/golang/protobuf/ptypes/empty google.golang.org/genproto/googleapis/type/date google.golang.org/genproto/protobuf/field_mask google.golang.org/api/iterator golang.org/x/net/context/ctxhttp golang.org/x/oauth2/jws golang.org/x/oauth2/internal cloud.google.com/go/internal/trace os/user go.opencensus.io/resource golang.org/x/oauth2 go.opencensus.io/metric/metricdata runtime/pprof golang.org/x/oauth2/authhandler golang.org/x/oauth2/google/internal/externalaccount golang.org/x/oauth2/jwt google.golang.org/grpc/internal/transport google.golang.org/api/internal/impersonate go.opencensus.io/tag go.opencensus.io/metric/metricproducer go.opencensus.io/trace/propagation go.opencensus.io/stats/internal encoding/gob go.opencensus.io/stats go.opencensus.io/stats/view go.opencensus.io/plugin/ocgrpc github.com/google/go-cmp/cmp/internal/diff github.com/google/go-cmp/cmp/internal/function golang.org/x/oauth2/google github.com/googleapis/enterprise-certificate-proxy/client/util net/rpc github.com/google/go-cmp/cmp/internal/value google.golang.org/grpc google.golang.org/grpc/credentials/alts/internal github.com/google/go-cmp/cmp google.golang.org/grpc/credentials/alts/internal/conn google.golang.org/grpc/internal/googlecloud github.com/googleapis/enterprise-certificate-proxy/client google.golang.org/grpc/credentials/oauth google.golang.org/api/transport/cert go.opencensus.io/plugin/ochttp/propagation/b3 google.golang.org/api/googleapi/transport google.golang.org/api/transport/http/internal/propagation go.opencensus.io/plugin/ochttp github.com/prometheus/common/version github.com/gogo/protobuf/proto github.com/golang/snappy github.com/grafana/dskit/services github.com/sirupsen/logrus github.com/googleapis/gax-go/v2 google.golang.org/genproto/googleapis/iam/v1 cloud.google.com/go/internal google.golang.org/api/internal google.golang.org/grpc/balancer/grpclb/grpc_lb_v1 cloud.google.com/go/iam cloud.google.com/go/storage/internal/apiv2/stubs google.golang.org/api/option google.golang.org/api/transport/internal/dca google.golang.org/api/option/internaloption google.golang.org/grpc/balancer/grpclb google.golang.org/grpc/credentials/alts/internal/proto/grpc_gcp google.golang.org/grpc/credentials/alts/internal/handshaker/service google.golang.org/api/internal/gensupport google.golang.org/grpc/credentials/alts/internal/authinfo google.golang.org/grpc/credentials/alts/internal/handshaker google.golang.org/api/transport/http google.golang.org/grpc/credentials/alts google.golang.org/api/iamcredentials/v1 google.golang.org/grpc/credentials/google google.golang.org/api/transport/grpc google.golang.org/api/storage/v1 github.com/weaveworks/promrus golang.org/x/net/context github.com/gorilla/mux github.com/weaveworks/common/logging cloud.google.com/go/storage/internal/apiv2 google.golang.org/api/transport github.com/weaveworks/common/user github.com/grpc-ecosystem/go-grpc-middleware github.com/grafana/dskit/tenant github.com/opentracing/opentracing-go/ext github.com/prometheus/client_golang/prometheus/promhttp github.com/opentracing-contrib/go-grpc github.com/jpillora/backoff golang.org/x/oauth2/clientcredentials github.com/mwitkow/go-conntrack golang.org/x/crypto/blowfish golang.org/x/crypto/bcrypt github.com/prometheus/common/config github.com/gogo/protobuf/sortkeys github.com/gogo/protobuf/types github.com/prometheus/exporter-toolkit/web github.com/sercand/kuberesolver cloud.google.com/go/storage github.com/felixge/httpsnoop github.com/opentracing-contrib/go-stdlib/nethttp github.com/weaveworks/common/grpc github.com/uber/jaeger-client-go/internal/spanlog github.com/uber/jaeger-client-go/log hash/adler32 github.com/uber/jaeger-lib/metrics compress/zlib go.uber.org/atomic github.com/uber/jaeger-client-go/thrift github.com/uber/jaeger-lib/metrics/prometheus net/http/internal/testcert net/http/httptest os/signal github.com/gogo/googleapis/google/rpc github.com/weaveworks/common/signals golang.org/x/net/netutil github.com/thanos-io/objstore/providers/gcs internal/profile github.com/uber/jaeger-client-go/thrift-gen/jaeger github.com/grafana/mimir/pkg/storage/bucket/gcs github.com/uber/jaeger-client-go/thrift-gen/sampling github.com/gogo/status github.com/uber/jaeger-client-go/thrift-gen/zipkincore github.com/uber/jaeger-client-go/thrift-gen/baggage net/http/pprof github.com/weaveworks/common/httpgrpc github.com/grafana/regexp/syntax github.com/modern-go/concurrent github.com/modern-go/reflect2 github.com/uber/jaeger-client-go/thrift-gen/agent golang.org/x/crypto/blake2b github.com/uber/jaeger-client-go/utils golang.org/x/crypto/argon2 github.com/aws/aws-sdk-go-v2/internal/rand github.com/aws/aws-sdk-go-v2/internal/sdk github.com/aws/smithy-go/logging github.com/grafana/regexp github.com/uber/jaeger-client-go github.com/aws/smithy-go/middleware github.com/prometheus/prometheus/model/labels github.com/aws/smithy-go/ptr github.com/aws/aws-sdk-go-v2/aws github.com/aws/smithy-go/rand github.com/json-iterator/go github.com/aws/smithy-go github.com/aws/smithy-go/time github.com/aws/smithy-go/transport/http/internal/io github.com/aws/smithy-go/transport/http github.com/aws/aws-sdk-go-v2/credentials github.com/aws/aws-sdk-go-v2/aws/ratelimit github.com/uber/jaeger-client-go/internal/baggage/remote github.com/uber/jaeger-client-go/internal/throttler/remote github.com/uber/jaeger-client-go/rpcmetrics github.com/uber/jaeger-client-go/transport github.com/aws/aws-sdk-go-v2/aws/middleware github.com/aws/aws-sdk-go-v2/internal/timeconv github.com/aws/aws-sdk-go-v2/feature/ec2/imds/internal/config github.com/aws/smithy-go/io github.com/uber/jaeger-client-go/config github.com/aws/aws-sdk-go-v2/credentials/processcreds github.com/aws/aws-sdk-go-v2/aws/transport/http github.com/aws/aws-sdk-go-v2/aws/retry github.com/aws/aws-sdk-go-v2/aws/defaults github.com/weaveworks/common/tracing github.com/aws/aws-sdk-go-v2/aws/protocol/restjson github.com/weaveworks/common/instrument github.com/aws/aws-sdk-go-v2/internal/strings github.com/aws/aws-sdk-go-v2/aws/signer/internal/v4 github.com/weaveworks/common/middleware github.com/aws/aws-sdk-go-v2/feature/ec2/imds github.com/aws/aws-sdk-go-v2/credentials/endpointcreds/internal/client github.com/aws/aws-sdk-go-v2/credentials/endpointcreds github.com/aws/aws-sdk-go-v2/credentials/ec2rolecreds github.com/aws/smithy-go/encoding/httpbinding github.com/minio/minio-go/v7/pkg/encrypt github.com/aws/aws-sdk-go-v2/internal/configsources github.com/weaveworks/common/httpgrpc/server github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 github.com/aws/smithy-go/document github.com/aws/aws-sdk-go-v2/aws/signer/v4 github.com/aws/aws-sdk-go-v2/service/sso/internal/endpoints github.com/aws/aws-sdk-go-v2/service/sso/types github.com/aws/aws-sdk-go-v2/aws/protocol/query github.com/aws/aws-sdk-go-v2/aws/protocol/xml github.com/weaveworks/common/server github.com/aws/aws-sdk-go-v2/service/sts/internal/endpoints github.com/aws/aws-sdk-go-v2/service/sso github.com/aws/aws-sdk-go-v2/service/internal/presigned-url github.com/aws/aws-sdk-go-v2/service/sts/types github.com/aws/smithy-go/encoding github.com/aws/smithy-go/encoding/xml github.com/aws/aws-sdk-go-v2/internal/ini github.com/grafana/mimir/pkg/util/log github.com/aws/aws-sdk-go-v2/service/sts github.com/grafana/mimir/pkg/util archive/tar github.com/aws/aws-sdk-go-v2/credentials/ssocreds github.com/klauspost/compress/s2 github.com/minio/md5-simd github.com/minio/minio-go/v7/pkg/s3utils github.com/klauspost/cpuid/v2 gopkg.in/ini.v1 github.com/minio/minio-go/v7/pkg/lifecycle github.com/aws/aws-sdk-go-v2/credentials/stscreds github.com/aws/aws-sdk-go-v2/config github.com/minio/minio-go/v7/pkg/set github.com/minio/minio-go/v7/pkg/notification github.com/minio/sha256-simd github.com/minio/minio-go/v7/pkg/signer github.com/rs/xid github.com/minio/minio-go/v7/pkg/sse github.com/minio/minio-go/v7/pkg/tags github.com/minio/minio-go/v7/pkg/replication net/http/cookiejar github.com/minio/minio-go/v7/pkg/credentials github.com/ncw/swift golang.org/x/net/publicsuffix github.com/stretchr/objx github.com/stretchr/testify/assert github.com/minio/minio-go/v7 github.com/thanos-io/objstore/providers/swift github.com/bradfitz/gomemcache/memcache github.com/grafana/mimir/pkg/storage/bucket/swift github.com/cespare/xxhash github.com/facette/natsort github.com/grafana/dskit/dns/godns encoding/base32 github.com/stretchr/testify/mock golang.org/x/net/bpf golang.org/x/net/internal/socket github.com/grafana/dskit/multierror github.com/prometheus/prometheus/util/gate github.com/grafana/dskit/spanlogger github.com/grafana/mimir/pkg/util/gate github.com/hashicorp/golang-lru/simplelru golang.org/x/net/ipv4 golang.org/x/net/ipv6 github.com/grafana/mimir/pkg/util/spanlogger github.com/gogo/protobuf/protoc-gen-gogo/descriptor github.com/grafana/dskit/backoff github.com/grafana/dskit/crypto/tls github.com/miekg/dns github.com/grafana/dskit/grpcencoding/snappy golang.org/x/time/rate google.golang.org/grpc/encoding/gzip github.com/thanos-io/objstore/providers/s3 github.com/grafana/dskit/grpcclient github.com/gogo/protobuf/gogoproto github.com/prometheus/prometheus/model/exemplar github.com/grafana/mimir/pkg/storage/bucket/s3 github.com/prometheus/prometheus/model/textparse container/heap github.com/edsrzf/mmap-go github.com/grafana/mimir/pkg/storage/bucket github.com/prometheus/prometheus/model/timestamp github.com/prometheus/prometheus/tsdb/chunkenc github.com/prometheus/prometheus/tsdb/errors github.com/prometheus/prometheus/model/metadata github.com/prometheus/client_golang/prometheus/testutil/promlint github.com/prometheus/prometheus/tsdb/fileutil github.com/stretchr/testify/require github.com/prometheus/prometheus/tsdb/chunks github.com/prometheus/client_golang/prometheus/testutil github.com/prometheus/prometheus/util/strutil github.com/oklog/ulid github.com/aws/aws-sdk-go/aws/awserr github.com/prometheus/prometheus/util/testutil github.com/aws/aws-sdk-go/internal/ini github.com/aws/aws-sdk-go/internal/shareddefaults github.com/aws/aws-sdk-go/internal/sync/singleflight github.com/prometheus/prometheus/tsdb/tsdbutil github.com/aws/aws-sdk-go/aws/endpoints github.com/prometheus/prometheus/storage github.com/aws/aws-sdk-go/aws/credentials github.com/aws/aws-sdk-go/internal/sdkio github.com/jmespath/go-jmespath github.com/prometheus/prometheus/promql/parser github.com/aws/aws-sdk-go/aws/awsutil github.com/aws/aws-sdk-go/internal/sdkrand github.com/aws/aws-sdk-go/internal/strings github.com/aws/aws-sdk-go/aws/credentials/processcreds github.com/aws/aws-sdk-go/internal/sdkuri github.com/prometheus/prometheus/discovery/targetgroup github.com/prometheus/prometheus/discovery github.com/prometheus/prometheus/model/relabel github.com/grafana/dskit/dns/miekgdns github.com/prometheus/prometheus/tsdb/encoding github.com/prometheus/prometheus/tsdb/hashcache github.com/grafana/dskit/dns github.com/prometheus/prometheus/tsdb/tombstones github.com/prometheus/prometheus/tsdb/index github.com/prometheus/prometheus/tsdb/record github.com/grafana/mimir/pkg/cacheutil github.com/prometheus/prometheus/tsdb/wal github.com/grafana/mimir/pkg/cache golang.org/x/sync/semaphore github.com/go-logr/logr go.opentelemetry.io/otel/attribute github.com/go-logr/logr/funcr go.opentelemetry.io/otel/codes go.opentelemetry.io/otel/internal/baggage go.opentelemetry.io/otel/baggage go.opentelemetry.io/otel/trace github.com/go-logr/stdr github.com/prometheus/prometheus/util/jsonutil google.golang.org/grpc/health/grpc_health_v1 github.com/golang/groupcache/singleflight github.com/grafana/dskit/runutil go.opentelemetry.io/otel/propagation github.com/grafana/mimir/pkg/util/extprom github.com/grafana/mimir/pkg/storage/tsdb/bucketcache github.com/grafana/mimir/pkg/storage/sharding go.opentelemetry.io/otel/internal/global github.com/grafana/mimir/pkg/storegateway/indexcache go.opentelemetry.io/otel github.com/prometheus/prometheus/util/stats github.com/grafana/mimir/pkg/storegateway/indexheader/fileutil embed github.com/grafana/dskit/grpcutil github.com/grafana/dskit/kv/codec github.com/hashicorp/go-cleanhttp github.com/mattn/go-isatty github.com/hashicorp/go-rootcerts github.com/hashicorp/go-immutable-radix github.com/mattn/go-colorable github.com/fatih/color github.com/mitchellh/mapstructure github.com/hashicorp/go-hclog github.com/armon/go-metrics go.etcd.io/etcd/api/v3/authpb go.etcd.io/etcd/api/v3/membershippb github.com/hashicorp/serf/coordinate go.etcd.io/etcd/api/v3/mvccpb github.com/hashicorp/consul/api go.uber.org/multierr go.etcd.io/etcd/api/v3/etcdserverpb go.uber.org/zap/buffer go.uber.org/zap/internal/bufferpool go.uber.org/zap/internal/color go.uber.org/zap/internal/exit go.uber.org/zap/zapcore go.uber.org/zap go.etcd.io/etcd/client/pkg/v3/fileutil github.com/grafana/dskit/kv/consul go.etcd.io/etcd/client/pkg/v3/tlsutil go.etcd.io/etcd/api/v3/v3rpc/rpctypes go.etcd.io/etcd/client/pkg/v3/transport github.com/coreos/go-semver/semver go.etcd.io/etcd/api/v3/version github.com/coreos/go-systemd/v22/journal go.etcd.io/etcd/client/pkg/v3/systemd go.etcd.io/etcd/client/pkg/v3/logutil go.etcd.io/etcd/client/pkg/v3/types go.etcd.io/etcd/client/v3/credentials go.etcd.io/etcd/client/v3/internal/endpoint github.com/aws/aws-sdk-go/aws google.golang.org/grpc/resolver/manual go.uber.org/zap/zapgrpc go.etcd.io/etcd/client/v3/internal/resolver github.com/prometheus/client_golang/prometheus/push github.com/hashicorp/go-sockaddr github.com/armon/go-metrics/prometheus github.com/aws/aws-sdk-go/aws/request compress/lzw github.com/google/btree github.com/hashicorp/go-msgpack/codec github.com/aws/aws-sdk-go/aws/client github.com/aws/aws-sdk-go/private/protocol github.com/aws/aws-sdk-go/aws/corehandlers github.com/aws/aws-sdk-go/aws/csm github.com/aws/aws-sdk-go/private/protocol/rest github.com/aws/aws-sdk-go/private/protocol/query/queryutil github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil github.com/aws/aws-sdk-go/aws/signer/v4 github.com/aws/aws-sdk-go/private/protocol/json/jsonutil github.com/aws/aws-sdk-go/aws/ec2metadata github.com/aws/aws-sdk-go/private/protocol/query github.com/aws/aws-sdk-go/private/protocol/jsonrpc github.com/aws/aws-sdk-go/aws/credentials/endpointcreds github.com/aws/aws-sdk-go/service/sts github.com/aws/aws-sdk-go/private/protocol/restjson github.com/aws/aws-sdk-go/aws/credentials/ec2rolecreds go.etcd.io/etcd/client/v3 github.com/aws/aws-sdk-go/service/sso github.com/aws/aws-sdk-go/aws/defaults github.com/hashicorp/errwrap github.com/hashicorp/go-multierror github.com/aws/aws-sdk-go/service/sts/stsiface github.com/aws/aws-sdk-go/service/sso/ssoiface github.com/sean-/seed github.com/aws/aws-sdk-go/aws/credentials/stscreds github.com/aws/aws-sdk-go/aws/credentials/ssocreds github.com/hashicorp/memberlist github.com/grafana/dskit/modules github.com/aws/aws-sdk-go/aws/session github.com/grafana/dskit/netutil github.com/grafana/dskit/ring/shard github.com/grafana/dskit/runtimeconfig github.com/grafana/dskit/concurrency github.com/prometheus/common/sigv4 github.com/prometheus/alertmanager/cluster/clusterpb github.com/prometheus/prometheus/config github.com/grafana/mimir/pkg/alertmanager/alertmanagerpb github.com/grafana/dskit/kv/etcd github.com/prometheus/prometheus/tsdb github.com/grafana/dskit/kv/memberlist github.com/grafana/mimir/pkg/alertmanager/alertspb github.com/prometheus/alertmanager/pkg/labels github.com/grafana/mimir/pkg/alertmanager/alertstore/bucketclient github.com/prometheus/alertmanager/timeinterval github.com/josharian/intern github.com/mailru/easyjson/jlexer github.com/prometheus/alertmanager/config github.com/mailru/easyjson/buffer github.com/mailru/easyjson/jwriter github.com/grafana/dskit/kv github.com/go-openapi/swag github.com/grafana/mimir/pkg/alertmanager/alertstore/local github.com/grafana/mimir/pkg/alertmanager/alertstore github.com/grafana/dskit/ring github.com/prometheus/alertmanager/api/metrics github.com/hashicorp/golang-lru github.com/cenkalti/backoff/v4 github.com/oklog/run github.com/prometheus/alertmanager/cluster github.com/prometheus/alertmanager/types github.com/prometheus/alertmanager/provider github.com/prometheus/alertmanager/store github.com/prometheus/alertmanager/inhibit github.com/prometheus/alertmanager/nflog/nflogpb github.com/benbjohnson/clock github.com/prometheus/alertmanager/nflog github.com/gofrs/uuid github.com/prometheus/alertmanager/silence/silencepb github.com/shurcooL/httpfs/vfsutil github.com/grafana/dskit/ring/client github.com/shurcooL/vfsgen github.com/prometheus/alertmanager/silence golang.org/x/text/internal/tag github.com/prometheus/alertmanager/asset golang.org/x/text/internal/language github.com/prometheus/prometheus/util/teststorage github.com/grafana/mimir/pkg/storage/tsdb/metadata github.com/prometheus/prometheus/promql github.com/julienschmidt/httprouter github.com/grafana/mimir/pkg/storage/tsdb/block github.com/prometheus/common/route golang.org/x/text/internal/language/compact github.com/go-openapi/analysis/internal/debug github.com/go-openapi/jsonpointer golang.org/x/text/language github.com/go-openapi/jsonreference/internal github.com/go-openapi/jsonreference github.com/grafana/mimir/pkg/storegateway/indexheader github.com/go-openapi/spec golang.org/x/text/internal golang.org/x/text/cases github.com/asaskevich/govalidator github.com/grafana/mimir/pkg/mimirpb github.com/prometheus/alertmanager/template github.com/prometheus/alertmanager/notify github.com/go-openapi/errors github.com/go-openapi/analysis/internal/flatten/normalize github.com/go-openapi/analysis/internal/flatten/operations github.com/go-openapi/analysis/internal/flatten/replace github.com/go-openapi/analysis/internal/flatten/schutils github.com/go-openapi/analysis/internal/flatten/sortref github.com/grafana/mimir/pkg/ingester/client github.com/prometheus/alertmanager/dispatch go.mongodb.org/mongo-driver/bson/primitive net/mail go.mongodb.org/mongo-driver/x/bsonx/bsoncore github.com/prometheus/alertmanager/api/v1 encoding/csv github.com/go-openapi/runtime/logger github.com/go-openapi/runtime/middleware/denco github.com/go-openapi/runtime/middleware/header github.com/docker/go-units github.com/jessevdk/go-flags github.com/go-openapi/runtime/flagext github.com/rs/cors go.mongodb.org/mongo-driver/bson/bsonrw github.com/grafana/mimir/pkg/util/net github.com/prometheus/alertmanager/notify/discord net/smtp github.com/prometheus/alertmanager/notify/email github.com/prometheus/alertmanager/notify/opsgenie github.com/prometheus/alertmanager/notify/pagerduty github.com/prometheus/alertmanager/notify/pushover github.com/prometheus/alertmanager/notify/slack github.com/grafana/mimir/pkg/storage/tsdb github.com/aws/aws-sdk-go/service/sns go.mongodb.org/mongo-driver/bson/bsoncodec gopkg.in/telebot.v3 github.com/grafana/mimir/cmd/metaconvert github.com/prometheus/alertmanager/notify/telegram github.com/prometheus/alertmanager/notify/victorops github.com/prometheus/alertmanager/notify/sns go.mongodb.org/mongo-driver/bson github.com/prometheus/alertmanager/notify/webhook github.com/prometheus/alertmanager/notify/wechat github.com/prometheus/alertmanager/provider/mem github.com/prometheus/alertmanager/ui github.com/google/pprof/profile github.com/go-openapi/strfmt github.com/grafana/mimir/pkg/storage/tsdb/bucketindex github.com/grafana/mimir/pkg/ingester/activeseries github.com/grafana/mimir/pkg/util/extract github.com/grafana/mimir/pkg/util/globalerror github.com/grafana/mimir/pkg/util/validation github.com/felixge/fgprof github.com/go-openapi/analysis github.com/go-openapi/runtime github.com/go-openapi/runtime/security github.com/grafana/mimir/pkg/compactor github.com/grafana/dskit/limiter github.com/go-openapi/loads github.com/grafana/mimir/pkg/distributor/forwarding github.com/go-openapi/runtime/middleware/untyped github.com/go-openapi/validate github.com/grafana/mimir/pkg/querier/stats github.com/grafana/mimir/pkg/util/httpgrpcutil github.com/grafana/mimir/pkg/util/limiter github.com/grafana/mimir/pkg/util/math github.com/prometheus/prometheus/prompb github.com/gogo/protobuf/jsonpb github.com/go-openapi/runtime/middleware github.com/prometheus/alertmanager/api/v2/models go.opentelemetry.io/collector/pdata/internal/data go.opentelemetry.io/collector/pdata/internal/data/protogen/common/v1 go.opentelemetry.io/collector/service/featuregate github.com/prometheus/prometheus/util/osutil github.com/prometheus/alertmanager/api/v2/restapi/operations/alert github.com/prometheus/alertmanager/api/v2/restapi/operations/alertgroup github.com/prometheus/alertmanager/api/v2/restapi/operations/general github.com/prometheus/alertmanager/api/v2/restapi/operations/receiver github.com/prometheus/alertmanager/api/v2/restapi/operations/silence go.opentelemetry.io/collector/pdata/internal/data/protogen/resource/v1 github.com/prometheus/prometheus/util/pool github.com/prometheus/prometheus/scrape go.opentelemetry.io/collector/pdata/internal/data/protogen/logs/v1 github.com/prometheus/alertmanager/api/v2/restapi/operations github.com/prometheus/alertmanager/api/v2/restapi go.opentelemetry.io/collector/pdata/internal/data/protogen/collector/logs/v1 github.com/prometheus/alertmanager/api/v2 go.opentelemetry.io/collector/pdata/internal/data/protogen/metrics/v1 go.opentelemetry.io/collector/pdata/internal/data/protogen/trace/v1 github.com/grafana/mimir/pkg/alertmanager/merger github.com/prometheus/alertmanager/api go.opentelemetry.io/collector/pdata/internal/data/protogen/collector/trace/v1 github.com/grafana/mimir/pkg/alertmanager go.opentelemetry.io/collector/pdata/internal/data/protogen/collector/metrics/v1 go.opentelemetry.io/collector/pdata/internal/otlp github.com/weaveworks/common/mtime github.com/grafana/mimir/pkg/distributor/distributorpb github.com/grafana/mimir/pkg/frontend/v1/frontendv1pb go.opentelemetry.io/collector/pdata/internal github.com/grafana/mimir/pkg/scheduler/queue github.com/grafana/mimir/pkg/frontend/v1 github.com/grafana/mimir/pkg/frontend/v2/frontendv2pb github.com/grafana/mimir/pkg/util/servicediscovery github.com/grafana/mimir/pkg/scheduler/schedulerdiscovery github.com/grafana/mimir/pkg/scheduler/schedulerpb github.com/grafana/mimir/pkg/storage/chunk github.com/grafana/mimir/pkg/querier/batch github.com/grafana/mimir/pkg/util/activitytracker github.com/grafana/mimir/pkg/frontend/v2 github.com/grafana/mimir/pkg/querier/engine go.opentelemetry.io/collector/pdata/plog go.opentelemetry.io/collector/pdata/pmetric go.opentelemetry.io/collector/pdata/ptrace go.opentelemetry.io/collector/pdata/pcommon go.opentelemetry.io/collector/pdata/pmetric/pmetricotlp github.com/grafana/mimir/pkg/querier/iterators github.com/grafana/mimir/pkg/storage/lazyquery go.opentelemetry.io/collector/consumer/consumererror github.com/grafana/mimir/pkg/storage/series github.com/grafana/mimir/pkg/storegateway/storepb github.com/dustin/go-humanize github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/prometheusremotewrite github.com/grafana/mimir/pkg/util/listblocks github.com/grafana/mimir/pkg/util/pool github.com/grafana/mimir/pkg/util/push github.com/grafana/mimir/pkg/util/chunkcompat github.com/prometheus/prometheus/template github.com/grafana/mimir/pkg/distributor github.com/prometheus/prometheus/model/rulefmt github.com/prometheus/prometheus/notifier github.com/grafana/mimir/pkg/storegateway/hintspb github.com/grafana/mimir/pkg/storegateway/storegatewaypb github.com/prometheus/prometheus/rules github.com/grafana/mimir/pkg/storegateway github.com/prometheus/prometheus/util/logging go.opentelemetry.io/otel/metric/instrument/asyncfloat64 go.opentelemetry.io/otel/metric/instrument/asyncint64 go.opentelemetry.io/otel/metric/instrument/syncfloat64 go.opentelemetry.io/otel/metric/instrument/syncint64 go.opentelemetry.io/otel/semconv/internal github.com/grafana/mimir/pkg/ruler/rulespb go.opentelemetry.io/otel/metric go.opentelemetry.io/otel/semconv/v1.12.0 go.opentelemetry.io/otel/metric/internal/global github.com/grafana/mimir/pkg/ruler/rulestore/local go.opentelemetry.io/otel/metric/global github.com/grafana/mimir/pkg/ruler/rulestore github.com/grafana/mimir/pkg/util/version go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp github.com/grafana/mimir/pkg/ruler/rulestore/bucketclient github.com/prometheus/client_golang/api github.com/prometheus/client_golang/api/prometheus/v1 github.com/prometheus/prometheus/discovery/refresh github.com/prometheus/prometheus/storage/remote github.com/spf13/afero/mem github.com/spf13/afero github.com/grafana/mimir/pkg/scheduler github.com/prometheus/prometheus/util/httputil github.com/grafana/mimir/pkg/util/gziphandler github.com/grafana/mimir/pkg/shipper github.com/grafana/mimir/pkg/api/error github.com/grafana/mimir/pkg/frontend/querymiddleware/astmapper github.com/grafana/mimir/pkg/frontend/transport github.com/grafana/mimir/pkg/querier github.com/prometheus/prometheus/web/api/v1 github.com/grafana/mimir/pkg/frontend/querymiddleware github.com/grafana/mimir/pkg/util/instrumentation github.com/grafana/mimir/pkg/querier/worker github.com/grafana/mimir/pkg/usagestats github.com/grafana/mimir/pkg/util/fs github.com/grafana/mimir/pkg/util/noauth github.com/grafana/mimir/pkg/util/process github.com/grafana/mimir/pkg/util/fieldcategory github.com/grafana/mimir/pkg/ingester github.com/grafana/mimir/pkg/util/usage github.com/grafana/mimir/pkg/continuoustest github.com/grafana/mimir/pkg/ruler github.com/grafana/mimir/pkg/frontend github.com/grafana/mimir/pkg/querier/tenantfederation database/sql github.com/rainycape/unidecode github.com/gosimple/slug github.com/grafana-tools/sdk github.com/grafana/mimir/pkg/flusher github.com/prometheus/client_golang/prometheus/collectors github.com/grafana/mimir/pkg/mimirtool/minisdk github.com/grafana/mimir/pkg/mimirtool/rules/rwrulefmt github.com/grafana/mimir/cmd/mimir-continuous-test github.com/grafana/mimir/pkg/mimirtool/backfill github.com/grafana/mimir/pkg/mimirtool/analyze github.com/grafana/mimir/pkg/mimirtool/client github.com/dlclark/regexp2/syntax github.com/grafana/mimir/pkg/api github.com/mitchellh/colorstring github.com/grafana/mimir/pkg/mimirtool/rules github.com/alecthomas/template/parse github.com/dlclark/regexp2 github.com/grafana/mimir/pkg/mimir github.com/alecthomas/template go/scanner github.com/alecthomas/chroma go/ast github.com/alecthomas/chroma/formatters/html github.com/grafana/mimir/cmd/mimir github.com/grafana/mimir/tools/doc-generator/parse github.com/alecthomas/chroma/formatters/svg github.com/alecthomas/chroma/formatters github.com/grafana/mimir/pkg/mimirtool/config github.com/alecthomas/chroma/lexers/internal github.com/alecthomas/chroma/lexers/a github.com/alecthomas/chroma/lexers/b github.com/alecthomas/chroma/lexers/p github.com/alecthomas/chroma/lexers/j github.com/alecthomas/chroma/lexers/d github.com/alecthomas/chroma/lexers/c github.com/alecthomas/chroma/lexers/e github.com/alecthomas/chroma/lexers/f github.com/alecthomas/chroma/lexers/i github.com/alecthomas/chroma/lexers/h github.com/alecthomas/chroma/lexers/k github.com/alecthomas/chroma/lexers/l github.com/alecthomas/chroma/lexers/n github.com/alecthomas/chroma/lexers/o github.com/alecthomas/chroma/lexers/q github.com/alecthomas/chroma/lexers/r github.com/alecthomas/chroma/lexers/t github.com/alecthomas/chroma/lexers/circular github.com/alecthomas/chroma/lexers/g github.com/alecthomas/chroma/lexers/m github.com/alecthomas/chroma/lexers/s github.com/alecthomas/chroma/lexers/v github.com/alecthomas/chroma/lexers/w github.com/alecthomas/chroma/lexers/x github.com/alecthomas/chroma/lexers/y github.com/alecthomas/chroma/lexers/z github.com/alecthomas/chroma/styles go/doc/comment internal/lazyregexp github.com/google/go-querystring/query go/doc golang.org/x/crypto/openpgp/errors golang.org/x/crypto/openpgp/armor compress/bzip2 golang.org/x/crypto/cast5 github.com/alecthomas/chroma/lexers gopkg.in/alecthomas/kingpin.v2 golang.org/x/crypto/openpgp/elgamal github.com/alecthomas/chroma/quick golang.org/x/crypto/openpgp/s2k github.com/grafana/mimir/pkg/mimirtool/printer image github.com/grafana/mimir/tools/querytee image/internal/imageutil image/jpeg github.com/grafana/mimir/cmd/query-tee golang.org/x/crypto/openpgp/packet github.com/grafana/mimir/pkg/mimirtool/commands golang.org/x/crypto/openpgp github.com/google/go-github/v32/github github.com/grafana/mimir/pkg/mimirtool/version github.com/grafana/mimir/cmd/mimirtool ==> Starting check()... === RUN TestConvertTenantBlocks --- PASS: TestConvertTenantBlocks (0.01s) === RUN TestConvertTenantBlocksDryMode --- PASS: TestConvertTenantBlocksDryMode (0.01s) PASS ok github.com/grafana/mimir/cmd/metaconvert 0.339s === RUN TestFlagParsing === RUN TestFlagParsing/new_flag,_with_config === RUN TestFlagParsing/root_level_configuration_option_specified_as_an_empty_node_in_YAML_does_not_set_entire_config_to_zero_value === RUN TestFlagParsing/common_yaml_inheritance_with_common_config_in_the_first_place === RUN TestFlagParsing/common_flag_inheritance === RUN TestFlagParsing/config === RUN TestFlagParsing/common_yaml_unmarshaling_is_strict === RUN TestFlagParsing/help-short === RUN TestFlagParsing/config_with_expand-env === RUN TestFlagParsing/user_visible_module_listing === RUN TestFlagParsing/common_yaml_inheritance_with_common_config_in_the_last_place === RUN TestFlagParsing/default_values === RUN TestFlagParsing/help-all === RUN TestFlagParsing/config_with_arguments_override === RUN TestFlagParsing/user_visible_module_listing_flag_take_precedence_over_target_flag === RUN TestFlagParsing/version === RUN TestFlagParsing/help === RUN TestFlagParsing/unknown_flag === RUN TestFlagParsing/common_yaml_sets_a_value_but_specific_config_reverts_it_back === RUN TestFlagParsing/common_yaml_overridden_by_a_common_flag_and_specific_flag --- PASS: TestFlagParsing (2.93s) --- PASS: TestFlagParsing/new_flag,_with_config (0.14s) --- PASS: TestFlagParsing/root_level_configuration_option_specified_as_an_empty_node_in_YAML_does_not_set_entire_config_to_zero_value (0.22s) --- PASS: TestFlagParsing/common_yaml_inheritance_with_common_config_in_the_first_place (0.14s) --- PASS: TestFlagParsing/common_flag_inheritance (0.12s) --- PASS: TestFlagParsing/config (0.12s) --- PASS: TestFlagParsing/common_yaml_unmarshaling_is_strict (0.04s) --- PASS: TestFlagParsing/help-short (0.11s) --- PASS: TestFlagParsing/config_with_expand-env (0.09s) --- PASS: TestFlagParsing/user_visible_module_listing (0.12s) --- PASS: TestFlagParsing/common_yaml_inheritance_with_common_config_in_the_last_place (0.27s) --- PASS: TestFlagParsing/default_values (0.21s) --- PASS: TestFlagParsing/help-all (0.31s) --- PASS: TestFlagParsing/config_with_arguments_override (0.20s) --- PASS: TestFlagParsing/user_visible_module_listing_flag_take_precedence_over_target_flag (0.04s) --- PASS: TestFlagParsing/version (0.04s) --- PASS: TestFlagParsing/help (0.14s) --- PASS: TestFlagParsing/unknown_flag (0.20s) --- PASS: TestFlagParsing/common_yaml_sets_a_value_but_specific_config_reverts_it_back (0.23s) --- PASS: TestFlagParsing/common_yaml_overridden_by_a_common_flag_and_specific_flag (0.19s) === RUN TestHelp === RUN TestHelp/basic === RUN TestHelp/all --- PASS: TestHelp (0.26s) --- PASS: TestHelp/basic (0.16s) --- PASS: TestHelp/all (0.11s) === RUN TestExpandEnvironmentVariables === RUN TestExpandEnvironmentVariables/x$y === RUN TestExpandEnvironmentVariables/x${y} === RUN TestExpandEnvironmentVariables/x$Y === RUN TestExpandEnvironmentVariables/x${Y} === RUN TestExpandEnvironmentVariables/x${Z:D} === RUN TestExpandEnvironmentVariables/x${Z:A_B_C_D} === RUN TestExpandEnvironmentVariables/x${Z:} === RUN TestExpandEnvironmentVariables/x$y:D --- PASS: TestExpandEnvironmentVariables (0.00s) --- PASS: TestExpandEnvironmentVariables/x$y (0.00s) --- PASS: TestExpandEnvironmentVariables/x${y} (0.00s) --- PASS: TestExpandEnvironmentVariables/x$Y (0.00s) --- PASS: TestExpandEnvironmentVariables/x${Y} (0.00s) --- PASS: TestExpandEnvironmentVariables/x${Z:D} (0.00s) --- PASS: TestExpandEnvironmentVariables/x${Z:A_B_C_D} (0.00s) --- PASS: TestExpandEnvironmentVariables/x${Z:} (0.00s) --- PASS: TestExpandEnvironmentVariables/x$y:D (0.00s) === RUN TestParseConfigFileParameter === RUN TestParseConfigFileParameter/#00 === RUN TestParseConfigFileParameter/--foo === RUN TestParseConfigFileParameter/-f_-a === RUN TestParseConfigFileParameter/--config.file=foo === RUN TestParseConfigFileParameter/--config.file_foo === RUN TestParseConfigFileParameter/--config.file=foo_--config.expand-env === RUN TestParseConfigFileParameter/--config.expand-env_--config.file=foo === RUN TestParseConfigFileParameter/--opt1_--config.file=foo === RUN TestParseConfigFileParameter/--opt1_--config.file_foo === RUN TestParseConfigFileParameter/--opt1_--config.file=foo_--config.expand-env === RUN TestParseConfigFileParameter/--opt1_--config.expand-env_--config.file=foo === RUN TestParseConfigFileParameter/--config.file=foo_--opt1 === RUN TestParseConfigFileParameter/--config.file_foo_--opt1 === RUN TestParseConfigFileParameter/--config.file=foo_--config.expand-env_--opt1 === RUN TestParseConfigFileParameter/--config.expand-env_--config.file=foo_--opt1 === RUN TestParseConfigFileParameter/--config.file=foo_--opt1_--config.expand-env === RUN TestParseConfigFileParameter/--config.expand-env_--opt1_--config.file=foo --- PASS: TestParseConfigFileParameter (0.01s) --- PASS: TestParseConfigFileParameter/#00 (0.00s) --- PASS: TestParseConfigFileParameter/--foo (0.00s) --- PASS: TestParseConfigFileParameter/-f_-a (0.00s) --- PASS: TestParseConfigFileParameter/--config.file=foo (0.00s) --- PASS: TestParseConfigFileParameter/--config.file_foo (0.00s) --- PASS: TestParseConfigFileParameter/--config.file=foo_--config.expand-env (0.00s) --- PASS: TestParseConfigFileParameter/--config.expand-env_--config.file=foo (0.00s) --- PASS: TestParseConfigFileParameter/--opt1_--config.file=foo (0.00s) --- PASS: TestParseConfigFileParameter/--opt1_--config.file_foo (0.00s) --- PASS: TestParseConfigFileParameter/--opt1_--config.file=foo_--config.expand-env (0.00s) --- PASS: TestParseConfigFileParameter/--opt1_--config.expand-env_--config.file=foo (0.00s) --- PASS: TestParseConfigFileParameter/--config.file=foo_--opt1 (0.00s) --- PASS: TestParseConfigFileParameter/--config.file_foo_--opt1 (0.00s) --- PASS: TestParseConfigFileParameter/--config.file=foo_--config.expand-env_--opt1 (0.00s) --- PASS: TestParseConfigFileParameter/--config.expand-env_--config.file=foo_--opt1 (0.00s) --- PASS: TestParseConfigFileParameter/--config.file=foo_--opt1_--config.expand-env (0.00s) --- PASS: TestParseConfigFileParameter/--config.expand-env_--opt1_--config.file=foo (0.00s) === RUN TestFieldCategoryOverridesNotStale --- PASS: TestFieldCategoryOverridesNotStale (0.08s) PASS ok github.com/grafana/mimir/cmd/mimir 4.006s ? github.com/grafana/mimir/cmd/mimir-continuous-test [no test files] ? github.com/grafana/mimir/cmd/mimirtool [no test files] === RUN TestMimirReadRoutes --- PASS: TestMimirReadRoutes (0.00s) PASS ok github.com/grafana/mimir/cmd/query-tee 0.323s ? github.com/grafana/mimir/integration [no test files] ? github.com/grafana/mimir/integration/ca [no test files] ? github.com/grafana/mimir/integration/e2emimir [no test files] === RUN TestResolveConfigs === RUN TestResolveConfigs/ResolveConfigs_annotates_values_that_match_defaults === RUN TestResolveConfigs/ResolveConfigs_sets_configuration_parameters_from_cli_arguments === RUN TestResolveConfigs/ResolveConfigs_sets_configuration_parameters_from_mounted_ConfigMaps === RUN TestResolveConfigs/ResolveConfigs_sets_configuration_parameters_from_mounted_Secrets --- PASS: TestResolveConfigs (1.35s) --- PASS: TestResolveConfigs/ResolveConfigs_annotates_values_that_match_defaults (0.37s) --- PASS: TestResolveConfigs/ResolveConfigs_sets_configuration_parameters_from_cli_arguments (0.36s) --- PASS: TestResolveConfigs/ResolveConfigs_sets_configuration_parameters_from_mounted_ConfigMaps (0.34s) --- PASS: TestResolveConfigs/ResolveConfigs_sets_configuration_parameters_from_mounted_Secrets (0.27s) PASS ok github.com/grafana/mimir/operations/compare-helm-with-jsonnet/plugins/resolve-config 1.876s === RUN TestMultitenantAlertmanager_GetStatusHandler --- PASS: TestMultitenantAlertmanager_GetStatusHandler (0.39s) === RUN TestAlertmanagerMetricsStore --- PASS: TestAlertmanagerMetricsStore (0.04s) === RUN TestAlertmanagerMetricsRemoval --- PASS: TestAlertmanagerMetricsRemoval (0.12s) === RUN TestIsHealthyForAlertmanagerOperations === PAUSE TestIsHealthyForAlertmanagerOperations === RUN TestDispatcherGroupLimits === RUN TestDispatcherGroupLimits/no_limit === RUN TestDispatcherGroupLimits/high_limit === RUN TestDispatcherGroupLimits/low_limit --- PASS: TestDispatcherGroupLimits (0.62s) --- PASS: TestDispatcherGroupLimits/no_limit (0.49s) --- PASS: TestDispatcherGroupLimits/high_limit (0.07s) --- PASS: TestDispatcherGroupLimits/low_limit (0.06s) === RUN TestDispatcherLoggerInsightKey --- PASS: TestDispatcherLoggerInsightKey (0.09s) === RUN TestAlertsLimiterWithNoLimits --- PASS: TestAlertsLimiterWithNoLimits (0.00s) === RUN TestAlertsLimiterWithCountLimit --- PASS: TestAlertsLimiterWithCountLimit (0.00s) === RUN TestAlertsLimiterWithSizeLimit --- PASS: TestAlertsLimiterWithSizeLimit (0.00s) === RUN TestAlertsLimiterWithSizeLimitAndAnnotationUpdate --- PASS: TestAlertsLimiterWithSizeLimitAndAnnotationUpdate (0.00s) === RUN TestAMConfigValidationAPI === RUN TestAMConfigValidationAPI/Should_return_error_if_the_alertmanager_config_contains_no_receivers === RUN TestAMConfigValidationAPI/Should_pass_if_the_alertmanager_config_is_valid === RUN TestAMConfigValidationAPI/Should_return_error_if_the_config_is_empty_due_to_wrong_indentation === RUN TestAMConfigValidationAPI/Should_return_error_if_the_alertmanager_config_is_empty_due_to_wrong_key === RUN TestAMConfigValidationAPI/Should_return_error_if_the_external_template_file_name_contains_an_absolute_path === RUN TestAMConfigValidationAPI/Should_return_error_if_the_external_template_file_name_contains_a_relative_path === RUN TestAMConfigValidationAPI/Should_return_error_if_the_external_template_file_name_is_not_a_valid_filename === RUN TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_the_root_/ === RUN TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_the_root_with_repeated_separators_/// === RUN TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_an_absolute_path === RUN TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_a_relative_path === RUN TestAMConfigValidationAPI/Should_pass_if_the_referenced_template_is_valid_filename === RUN TestAMConfigValidationAPI/Should_return_error_if_global_HTTP_password_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_HTTP_bearer_token_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_HTTP_credentials_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_OAuth2_client_secret_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_OAuth2_proxy_url_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_OAuth2_TLS_key_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_receiver's_HTTP_password_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_receiver's_HTTP_bearer_token_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_receiver's_HTTP_credentials_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_receiver's_OAuth2_client_secret_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_receiver's_OAuth2_proxy_url_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_slack_api_url_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_Slack_api_url_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_opsgenie_api_key_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_OpsGenie_api_key_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_victorops_api_key_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_VictorOps_api_key_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_PagerDuty_service_key_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_PagerDuty_routing_key_file_is_set === RUN TestAMConfigValidationAPI/should_return_error_if_template_is_wrong === RUN TestAMConfigValidationAPI/config_too_big === RUN TestAMConfigValidationAPI/config_size_OK === RUN TestAMConfigValidationAPI/templates_limit_reached === RUN TestAMConfigValidationAPI/templates_limit_not_reached === RUN TestAMConfigValidationAPI/template_size_limit_reached === RUN TestAMConfigValidationAPI/template_size_limit_ok --- PASS: TestAMConfigValidationAPI (0.26s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_alertmanager_config_contains_no_receivers (0.00s) --- PASS: TestAMConfigValidationAPI/Should_pass_if_the_alertmanager_config_is_valid (0.05s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_config_is_empty_due_to_wrong_indentation (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_alertmanager_config_is_empty_due_to_wrong_key (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_external_template_file_name_contains_an_absolute_path (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_external_template_file_name_contains_a_relative_path (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_external_template_file_name_is_not_a_valid_filename (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_the_root_/ (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_the_root_with_repeated_separators_/// (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_an_absolute_path (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_a_relative_path (0.00s) --- PASS: TestAMConfigValidationAPI/Should_pass_if_the_referenced_template_is_valid_filename (0.03s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_HTTP_password_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_HTTP_bearer_token_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_HTTP_credentials_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_OAuth2_client_secret_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_OAuth2_proxy_url_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_OAuth2_TLS_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_receiver's_HTTP_password_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_receiver's_HTTP_bearer_token_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_receiver's_HTTP_credentials_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_receiver's_OAuth2_client_secret_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_receiver's_OAuth2_proxy_url_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_slack_api_url_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_Slack_api_url_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_opsgenie_api_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_OpsGenie_api_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_victorops_api_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_VictorOps_api_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_PagerDuty_service_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_PagerDuty_routing_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/should_return_error_if_template_is_wrong (0.03s) --- PASS: TestAMConfigValidationAPI/config_too_big (0.00s) --- PASS: TestAMConfigValidationAPI/config_size_OK (0.03s) --- PASS: TestAMConfigValidationAPI/templates_limit_reached (0.00s) --- PASS: TestAMConfigValidationAPI/templates_limit_not_reached (0.03s) --- PASS: TestAMConfigValidationAPI/template_size_limit_reached (0.00s) --- PASS: TestAMConfigValidationAPI/template_size_limit_ok (0.03s) === RUN TestMultitenantAlertmanager_DeleteUserConfig --- PASS: TestMultitenantAlertmanager_DeleteUserConfig (0.00s) === RUN TestAMConfigListUserConfig --- PASS: TestAMConfigListUserConfig (0.33s) === RUN TestValidateAlertmanagerConfig === RUN TestValidateAlertmanagerConfig/TLSConfig === RUN TestValidateAlertmanagerConfig/*EmailConfig.AuthPasswordFile === RUN TestValidateAlertmanagerConfig/struct_containing_*HTTPClientConfig_as_direct_child === RUN TestValidateAlertmanagerConfig/struct_containing_*HTTPClientConfig_as_nested_child === RUN TestValidateAlertmanagerConfig/struct_containing_*HTTPClientConfig_as_nested_child_within_a_slice === RUN TestValidateAlertmanagerConfig/map_containing_*HTTPClientConfig === RUN TestValidateAlertmanagerConfig/map_containing_TLSConfig_as_nested_child === RUN TestValidateAlertmanagerConfig/*HTTPClientConfig === RUN TestValidateAlertmanagerConfig/*TLSConfig === RUN TestValidateAlertmanagerConfig/*GlobalConfig.SMTPAuthPasswordFile === RUN TestValidateAlertmanagerConfig/GlobalConfig.SMTPAuthPasswordFile === RUN TestValidateAlertmanagerConfig/EmailConfig.AuthPasswordFile === RUN TestValidateAlertmanagerConfig/HTTPClientConfig --- PASS: TestValidateAlertmanagerConfig (0.00s) --- PASS: TestValidateAlertmanagerConfig/TLSConfig (0.00s) --- PASS: TestValidateAlertmanagerConfig/*EmailConfig.AuthPasswordFile (0.00s) --- PASS: TestValidateAlertmanagerConfig/struct_containing_*HTTPClientConfig_as_direct_child (0.00s) --- PASS: TestValidateAlertmanagerConfig/struct_containing_*HTTPClientConfig_as_nested_child (0.00s) --- PASS: TestValidateAlertmanagerConfig/struct_containing_*HTTPClientConfig_as_nested_child_within_a_slice (0.00s) --- PASS: TestValidateAlertmanagerConfig/map_containing_*HTTPClientConfig (0.00s) --- PASS: TestValidateAlertmanagerConfig/map_containing_TLSConfig_as_nested_child (0.00s) --- PASS: TestValidateAlertmanagerConfig/*HTTPClientConfig (0.00s) --- PASS: TestValidateAlertmanagerConfig/*TLSConfig (0.00s) --- PASS: TestValidateAlertmanagerConfig/*GlobalConfig.SMTPAuthPasswordFile (0.00s) --- PASS: TestValidateAlertmanagerConfig/GlobalConfig.SMTPAuthPasswordFile (0.00s) --- PASS: TestValidateAlertmanagerConfig/EmailConfig.AuthPasswordFile (0.00s) --- PASS: TestValidateAlertmanagerConfig/HTTPClientConfig (0.00s) === RUN TestDistributor_DistributeRequest === RUN TestDistributor_DistributeRequest/Write_/alerts,_Simple_AM_request,_all_AM_healthy === RUN TestDistributor_DistributeRequest/Write_/alerts,_no_healthy_Alertmanagers === RUN TestDistributor_DistributeRequest/Write_/alerts,_1_healthy_Alertmanager_out_of_3 === RUN TestDistributor_DistributeRequest/Write_/alerts,_3_healthy_Alertmanagers_out_of_5 === RUN TestDistributor_DistributeRequest/Read_/v1/alerts_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Read_/v2/alerts_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Read_/v2/alerts/groups_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Write_/alerts/groups_not_supported === RUN TestDistributor_DistributeRequest/Read_/v1/silences_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Read_/v2/silences_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Write_/silences_is_sent_to_only_1_AM === RUN TestDistributor_DistributeRequest/Read_/v1/silence/id_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Read_/v2/silence/id_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Write_/silence/id_not_supported === RUN TestDistributor_DistributeRequest/Delete_/silence/id_is_sent_to_only_1_AM === RUN TestDistributor_DistributeRequest/Read_/status_is_sent_to_only_1_AM === RUN TestDistributor_DistributeRequest/Write_/status_not_supported === RUN TestDistributor_DistributeRequest/Read_/some/place/arbitrary_is_sent_to_only_1_AM === RUN TestDistributor_DistributeRequest/Read_/receivers_is_sent_to_only_1_AM === RUN TestDistributor_DistributeRequest/Write_/receivers_not_supported --- PASS: TestDistributor_DistributeRequest (0.17s) --- PASS: TestDistributor_DistributeRequest/Write_/alerts,_Simple_AM_request,_all_AM_healthy (0.01s) --- PASS: TestDistributor_DistributeRequest/Write_/alerts,_no_healthy_Alertmanagers (0.01s) --- PASS: TestDistributor_DistributeRequest/Write_/alerts,_1_healthy_Alertmanager_out_of_3 (0.00s) --- PASS: TestDistributor_DistributeRequest/Write_/alerts,_3_healthy_Alertmanagers_out_of_5 (0.01s) --- PASS: TestDistributor_DistributeRequest/Read_/v1/alerts_is_sent_to_3_AMs (0.02s) --- PASS: TestDistributor_DistributeRequest/Read_/v2/alerts_is_sent_to_3_AMs (0.02s) --- PASS: TestDistributor_DistributeRequest/Read_/v2/alerts/groups_is_sent_to_3_AMs (0.02s) --- PASS: TestDistributor_DistributeRequest/Write_/alerts/groups_not_supported (0.00s) --- PASS: TestDistributor_DistributeRequest/Read_/v1/silences_is_sent_to_3_AMs (0.02s) --- PASS: TestDistributor_DistributeRequest/Read_/v2/silences_is_sent_to_3_AMs (0.02s) --- PASS: TestDistributor_DistributeRequest/Write_/silences_is_sent_to_only_1_AM (0.00s) --- PASS: TestDistributor_DistributeRequest/Read_/v1/silence/id_is_sent_to_3_AMs (0.01s) --- PASS: TestDistributor_DistributeRequest/Read_/v2/silence/id_is_sent_to_3_AMs (0.02s) --- PASS: TestDistributor_DistributeRequest/Write_/silence/id_not_supported (0.00s) --- PASS: TestDistributor_DistributeRequest/Delete_/silence/id_is_sent_to_only_1_AM (0.00s) --- PASS: TestDistributor_DistributeRequest/Read_/status_is_sent_to_only_1_AM (0.00s) --- PASS: TestDistributor_DistributeRequest/Write_/status_not_supported (0.00s) --- PASS: TestDistributor_DistributeRequest/Read_/some/place/arbitrary_is_sent_to_only_1_AM (0.00s) --- PASS: TestDistributor_DistributeRequest/Read_/receivers_is_sent_to_only_1_AM (0.00s) --- PASS: TestDistributor_DistributeRequest/Write_/receivers_not_supported (0.00s) === RUN TestMultitenantAlertmanagerConfig_Validate === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_persistent_interval_is_negative === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_the_URL_just_contains_the_hostname_(because_it_can't_be_disambiguated_with_a_path) === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_with_empty_external_URL === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_persistent_interval_is_0 === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_external_URL_ends_with_/ === RUN TestMultitenantAlertmanagerConfig_Validate/should_succeed_if_external_URL_does_not_end_with_/ === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_external_URL_has_no_scheme === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_external_URL_has_no_hostname === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_zone_aware_is_enabled_but_zone_is_not_set === RUN TestMultitenantAlertmanagerConfig_Validate/should_pass_if_the_URL_just_contains_the_path_with_the_leading_/ === RUN TestMultitenantAlertmanagerConfig_Validate/should_pass_with_default_config --- PASS: TestMultitenantAlertmanagerConfig_Validate (0.02s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_persistent_interval_is_negative (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_the_URL_just_contains_the_hostname_(because_it_can't_be_disambiguated_with_a_path) (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_with_empty_external_URL (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_persistent_interval_is_0 (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_external_URL_ends_with_/ (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_succeed_if_external_URL_does_not_end_with_/ (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_external_URL_has_no_scheme (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_external_URL_has_no_hostname (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_zone_aware_is_enabled_but_zone_is_not_set (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_pass_if_the_URL_just_contains_the_path_with_the_leading_/ (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_pass_with_default_config (0.00s) === RUN TestMultitenantAlertmanager_loadAndSyncConfigs --- PASS: TestMultitenantAlertmanager_loadAndSyncConfigs (0.73s) === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=false === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=true === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=false === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=true === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=true === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=false multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=discord incident={}:{} level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=discord integration=discord[0] msg="Notify success" attempts=1 === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=false multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="ring doesn't exist in KV store yet" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=pagerduty incident={}:{} eventType=trigger level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=pagerduty integration=pagerduty[0] msg="Notify success" attempts=1 === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=false multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="ring doesn't exist in KV store yet" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=sns msg="SNS message successfully published" message_id=null sequencenumber=null level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=sns integration=sns[0] msg="Notify success" attempts=1 === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=true === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=true multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=opsgenie alert={}:{} level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=opsgenie integration=opsgenie[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"http://127.0.0.1:39249/v2/alerts\": dial tcp 127.0.0.1:39249: blocked address" level=debug component=MultiTenantAlertmanager user=user-1 integration=opsgenie alert={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=opsgenie alert={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=opsgenie alert={}:{} === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=true multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=pagerduty incident={}:{} eventType=trigger level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=pagerduty integration=pagerduty[0] msg="Notify attempt failed, will retry later" attempts=1 err="failed to post message to PagerDuty: Post \"http://127.0.0.1:37583\": dial tcp 127.0.0.1:37583: blocked address" level=debug component=MultiTenantAlertmanager user=user-1 integration=pagerduty incident={}:{} eventType=trigger level=debug component=MultiTenantAlertmanager user=user-1 integration=pagerduty incident={}:{} eventType=trigger level=debug component=MultiTenantAlertmanager user=user-1 integration=pagerduty incident={}:{} eventType=trigger === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=false === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=true === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=true multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=slack integration=slack[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"\": dial tcp 127.0.0.1:45083: blocked address" === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=true === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=false multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="ring doesn't exist in KV store yet" level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=telegram integration=telegram[0] msg="Notify attempt failed, will retry later" attempts=1 err="telebot: unexpected end of JSON input" === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=false multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="ring doesn't exist in KV store yet" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=webhook integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://www.google.com\": http: server gave HTTP response to HTTPS client" === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=false === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=true multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=discord incident={}:{} level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=discord integration=discord[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"\": dial tcp 127.0.0.1:35699: blocked address" level=debug component=MultiTenantAlertmanager user=user-1 integration=discord incident={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=discord incident={}:{} === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=true === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=false multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=webhook integration=webhook[0] msg="Notify success" attempts=1 === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=false multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="ring doesn't exist in KV store yet" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=opsgenie alert={}:{} level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=opsgenie integration=opsgenie[0] msg="Notify success" attempts=1 === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=false multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=wechat incident={}:{} level=error component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="wechat/wechat[0]: notify retry canceled due to unrecoverable error after 1 attempts: EOF" === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=true === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=true multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=telegram integration=telegram[0] msg="Notify attempt failed, will retry later" attempts=1 err="telebot: Post \"http://127.0.0.1:45347/botxxx/sendMessage\": dial tcp 127.0.0.1:45347: blocked address" === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=true === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=true multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="ring doesn't exist in KV store yet" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=webhook integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://www.google.com\": proxyconnect tcp: dial tcp 127.0.0.1:46759: blocked address" === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=false multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=slack integration=slack[0] msg="Notify success" attempts=1 === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=true multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="ring doesn't exist in KV store yet" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=webhook integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"http://127.0.0.1:41139\": dial tcp 127.0.0.1:41139: blocked address" === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=true multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=sns integration=sns[0] msg="Notify attempt failed, will retry later" attempts=1 err="RequestError: send request failed\ncaused by: Post \"http://127.0.0.1:37557/\": dial tcp 127.0.0.1:37557: blocked address" === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=true multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="ring doesn't exist in KV store yet" level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=wechat incident={}:{} level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=wechat integration=wechat[0] msg="Notify attempt failed, will retry later" attempts=1 err="Get \"\": dial tcp 127.0.0.1:33593: blocked address" level=debug component=MultiTenantAlertmanager user=user-1 integration=wechat incident={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=wechat incident={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=wechat incident={}:{} --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled (0.00s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=false (0.58s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=false (0.54s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=false (0.49s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=true (3.48s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=true (3.49s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=true (3.54s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=false (0.65s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=false (0.39s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=true (3.28s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=false (0.39s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=false (0.54s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=false (0.54s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=true (3.41s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=true (3.51s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=false (0.39s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=true (3.54s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=true (3.32s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=true (3.51s) === RUN TestMultitenantAlertmanager_deleteUnusedLocalUserState --- PASS: TestMultitenantAlertmanager_deleteUnusedLocalUserState (0.33s) === RUN TestMultitenantAlertmanager_zoneAwareSharding level=info component=MultiTenantAlertmanager msg="ring doesn't exist in KV store yet" level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=instance-1 ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=instance-2 ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=instance-3 ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user1 level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="Waiting for notification and silences to settle..." level=debug component=MultiTenantAlertmanager msg="contacting replica for full state" user=user1 addr=127.0.0.1-3:0 level=error component=MultiTenantAlertmanager msg="rpc reading state from replica failed" addr=127.0.0.1-3:0 user=user1 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup 127.0.0.1-3: no such host\"" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="unable to read state from other Alertmanager replicas; trying to read from storage" err="failed to read state from any replica" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=2 level=debug component=MultiTenantAlertmanager msg="setting config" user=user2 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user2 level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="Waiting for notification and silences to settle..." level=debug component=MultiTenantAlertmanager msg="contacting replica for full state" user=user2 addr=127.0.0.1-3:0 level=error component=MultiTenantAlertmanager msg="rpc reading state from replica failed" addr=127.0.0.1-3:0 user=user2 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup 127.0.0.1-3: no such host\"" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="unable to read state from other Alertmanager replicas; trying to read from storage" err="failed to read state from any replica" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="no state for user in storage; proceeding" level=debug component=MultiTenantAlertmanager msg="setting config" user=user3 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user3 level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="Waiting for notification and silences to settle..." level=debug component=MultiTenantAlertmanager msg="contacting replica for full state" user=user3 addr=127.0.0.1-3:0 level=error component=MultiTenantAlertmanager msg="rpc reading state from replica failed" addr=127.0.0.1-3:0 user=user3 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup 127.0.0.1-3: no such host\"" level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="unable to read state from other Alertmanager replicas; trying to read from storage" err="failed to read state from any replica" level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=3 level=debug component=MultiTenantAlertmanager msg="setting config" user=user1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user1 level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="Waiting for notification and silences to settle..." level=debug component=MultiTenantAlertmanager msg="contacting replica for full state" user=user1 addr=127.0.0.1-1:0 level=error component=MultiTenantAlertmanager msg="rpc reading state from replica failed" addr=127.0.0.1-1:0 user=user1 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup 127.0.0.1-1: no such host\"" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="unable to read state from other Alertmanager replicas; trying to read from storage" err="failed to read state from any replica" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="no state for user in storage; proceeding" level=debug component=MultiTenantAlertmanager msg="setting config" user=user2 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user2 level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="Waiting for notification and silences to settle..." level=debug component=MultiTenantAlertmanager msg="contacting replica for full state" user=user2 addr=127.0.0.1-2:0 level=debug component=MultiTenantAlertmanager msg="setting config" user=user3 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user3 level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="Waiting for notification and silences to settle..." level=debug component=MultiTenantAlertmanager msg="contacting replica for full state" user=user3 addr=127.0.0.1-2:0 level=error component=MultiTenantAlertmanager msg="rpc reading state from replica failed" addr=127.0.0.1-2:0 user=user2 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup 127.0.0.1-2: no such host\"" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="unable to read state from other Alertmanager replicas; trying to read from storage" err="failed to read state from any replica" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="no state for user in storage; proceeding" level=error component=MultiTenantAlertmanager msg="rpc reading state from replica failed" addr=127.0.0.1-2:0 user=user3 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup 127.0.0.1-2: no such host\"" level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="unable to read state from other Alertmanager replicas; trying to read from storage" err="failed to read state from any replica" level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="ring lifecycler is shutting down" ring=alertmanager level=info component=MultiTenantAlertmanager msg="unregistering instance from ring" ring=alertmanager level=info component=MultiTenantAlertmanager msg="instance removed from the ring" ring=alertmanager level=debug component=MultiTenantAlertmanager user=user1 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user1 component=nflog msg="Maintenance done" duration=577.999µs size=0 level=debug component=MultiTenantAlertmanager user=user1 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user1 component=silences msg="Maintenance done" duration=330µs size=0 level=debug component=MultiTenantAlertmanager user=user2 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user2 component=silences msg="Maintenance done" duration=382µs size=0 level=debug component=MultiTenantAlertmanager user=user2 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user2 component=nflog msg="Maintenance done" duration=1.100999ms size=0 level=debug component=MultiTenantAlertmanager user=user3 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user3 component=nflog msg="Maintenance done" duration=313µs size=0 level=debug component=MultiTenantAlertmanager user=user3 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user3 component=silences msg="Maintenance done" duration=317µs size=0 level=debug component=MultiTenantAlertmanager user=user2 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user2 component=nflog msg="Maintenance done" duration=442µs size=0 level=info component=MultiTenantAlertmanager msg="ring lifecycler is shutting down" ring=alertmanager level=debug component=MultiTenantAlertmanager user=user2 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user2 component=silences msg="Maintenance done" duration=482.999µs size=0 level=debug component=MultiTenantAlertmanager user=user3 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user3 component=nflog msg="Maintenance done" duration=312µs size=0 level=debug component=MultiTenantAlertmanager user=user3 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user3 component=silences msg="Maintenance done" duration=309µs size=0 level=info component=MultiTenantAlertmanager msg="unregistering instance from ring" ring=alertmanager level=info component=MultiTenantAlertmanager msg="instance removed from the ring" ring=alertmanager level=debug component=MultiTenantAlertmanager user=user1 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user1 component=nflog msg="Maintenance done" duration=461µs size=0 level=info component=MultiTenantAlertmanager msg="ring lifecycler is shutting down" ring=alertmanager level=info component=MultiTenantAlertmanager msg="unregistering instance from ring" ring=alertmanager level=info component=MultiTenantAlertmanager msg="instance removed from the ring" ring=alertmanager level=debug component=MultiTenantAlertmanager user=user1 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user1 component=silences msg="Maintenance done" duration=426µs size=0 --- PASS: TestMultitenantAlertmanager_zoneAwareSharding (1.71s) === RUN TestMultitenantAlertmanager_deleteUnusedRemoteUserState level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=instance-1 ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=instance-2 ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user2 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user2 level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user1 level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="no state for user in storage; proceeding" level=debug component=MultiTenantAlertmanager user=user2 msg="persisting state" user=user2 level=debug component=MultiTenantAlertmanager user=user1 msg="persisting state" user=user1 level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user2 level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user1 level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user2 level=info component=MultiTenantAlertmanager msg="deleted remote state for user" user=user1 level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="deactivating per-tenant alertmanager" user=user1 level=debug component=MultiTenantAlertmanager user=user1 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user1 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user1 component=nflog msg="Maintenance done" duration=536µs size=0 level=debug component=MultiTenantAlertmanager user=user1 component=silences msg="Maintenance done" duration=487µs size=0 level=info component=MultiTenantAlertmanager msg="deactivated per-tenant alertmanager" user=user1 level=info component=MultiTenantAlertmanager msg="deleted local directory for user" dir=/tmp/TestMultitenantAlertmanager_deleteUnusedRemoteUserState3651978769/002/user1 user=user1 level=info component=MultiTenantAlertmanager msg="ring lifecycler is shutting down" ring=alertmanager level=info component=MultiTenantAlertmanager msg="unregistering instance from ring" ring=alertmanager level=info component=MultiTenantAlertmanager msg="instance removed from the ring" ring=alertmanager level=debug component=MultiTenantAlertmanager user=user2 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user2 component=nflog msg="Maintenance done" duration=443.999µs size=0 level=info component=MultiTenantAlertmanager msg="ring lifecycler is shutting down" ring=alertmanager level=info component=MultiTenantAlertmanager msg="unregistering instance from ring" ring=alertmanager level=info component=MultiTenantAlertmanager msg="instance removed from the ring" ring=alertmanager level=debug component=MultiTenantAlertmanager user=user2 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user2 component=silences msg="Maintenance done" duration=346.999µs size=0 --- PASS: TestMultitenantAlertmanager_deleteUnusedRemoteUserState (0.90s) === RUN TestMultitenantAlertmanager_ServeHTTP --- PASS: TestMultitenantAlertmanager_ServeHTTP (0.21s) === RUN TestMultitenantAlertmanager_ServeHTTPWithFallbackConfig --- PASS: TestMultitenantAlertmanager_ServeHTTPWithFallbackConfig (0.44s) === RUN TestMultitenantAlertmanager_ServeHTTPBeforeSyncFailsIfConfigExists --- PASS: TestMultitenantAlertmanager_ServeHTTPBeforeSyncFailsIfConfigExists (0.26s) === RUN TestMultitenantAlertmanager_InitialSync === RUN TestMultitenantAlertmanager_InitialSync/with_no_instance_in_the_ring === RUN TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_PENDING_state_and_no_tokens === RUN TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_JOINING_state_and_some_tokens === RUN TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_ACTIVE_state_and_all_tokens === RUN TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_LEAVING_state_and_all_tokens --- PASS: TestMultitenantAlertmanager_InitialSync (1.04s) --- PASS: TestMultitenantAlertmanager_InitialSync/with_no_instance_in_the_ring (0.19s) --- PASS: TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_PENDING_state_and_no_tokens (0.17s) --- PASS: TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_JOINING_state_and_some_tokens (0.18s) --- PASS: TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_ACTIVE_state_and_all_tokens (0.20s) --- PASS: TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_LEAVING_state_and_all_tokens (0.30s) === RUN TestMultitenantAlertmanager_PerTenantSharding === RUN TestMultitenantAlertmanager_PerTenantSharding/1_instance,_RF_=_1 === RUN TestMultitenantAlertmanager_PerTenantSharding/2_instances,_RF_=_1 === RUN TestMultitenantAlertmanager_PerTenantSharding/3_instances,_RF_=_2 === RUN TestMultitenantAlertmanager_PerTenantSharding/5_instances,_RF_=_3 --- PASS: TestMultitenantAlertmanager_PerTenantSharding (10.62s) --- PASS: TestMultitenantAlertmanager_PerTenantSharding/1_instance,_RF_=_1 (1.27s) --- PASS: TestMultitenantAlertmanager_PerTenantSharding/2_instances,_RF_=_1 (1.78s) --- PASS: TestMultitenantAlertmanager_PerTenantSharding/3_instances,_RF_=_2 (2.91s) --- PASS: TestMultitenantAlertmanager_PerTenantSharding/5_instances,_RF_=_3 (4.66s) === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/when_an_instance_is_added_to_the_ring === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/when_an_instance_is_removed_from_the_ring === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_sync_when_an_instance_changes_state === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_sync_when_an_healthy_instance_becomes_unhealthy === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_sync_when_an_unhealthy_instance_becomes_healthy === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_NOT_sync_when_an_instance_updates_the_heartbeat === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_NOT_sync_when_an_instance_is_auto-forgotten_in_the_ring_but_was_already_unhealthy_in_the_previous_state --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges (1.98s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/when_an_instance_is_added_to_the_ring (0.31s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/when_an_instance_is_removed_from_the_ring (0.23s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_sync_when_an_instance_changes_state (0.29s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_sync_when_an_healthy_instance_becomes_unhealthy (0.24s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_sync_when_an_unhealthy_instance_becomes_healthy (0.35s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_NOT_sync_when_an_instance_updates_the_heartbeat (0.20s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_NOT_sync_when_an_instance_is_auto-forgotten_in_the_ring_but_was_already_unhealthy_in_the_previous_state (0.37s) === RUN TestMultitenantAlertmanager_RingLifecyclerShouldAutoForgetUnhealthyInstances --- PASS: TestMultitenantAlertmanager_RingLifecyclerShouldAutoForgetUnhealthyInstances (0.23s) === RUN TestMultitenantAlertmanager_InitialSyncFailure --- PASS: TestMultitenantAlertmanager_InitialSyncFailure (0.01s) === RUN TestAlertmanager_ReplicasPosition --- PASS: TestAlertmanager_ReplicasPosition (0.62s) === RUN TestAlertmanager_StateReplication === RUN TestAlertmanager_StateReplication/RF_=_1,_1_instance === RUN TestAlertmanager_StateReplication/RF_=_2,_2_instances === RUN TestAlertmanager_StateReplication/RF_=_3,_10_instance --- PASS: TestAlertmanager_StateReplication (13.35s) --- PASS: TestAlertmanager_StateReplication/RF_=_1,_1_instance (1.28s) --- PASS: TestAlertmanager_StateReplication/RF_=_2,_2_instances (2.38s) --- PASS: TestAlertmanager_StateReplication/RF_=_3,_10_instance (9.68s) === RUN TestAlertmanager_StateReplication_InitialSyncFromPeers === RUN TestAlertmanager_StateReplication_InitialSyncFromPeers/RF_=_2 === RUN TestAlertmanager_StateReplication_InitialSyncFromPeers/RF_=_3 --- PASS: TestAlertmanager_StateReplication_InitialSyncFromPeers (2.07s) --- PASS: TestAlertmanager_StateReplication_InitialSyncFromPeers/RF_=_2 (0.66s) --- PASS: TestAlertmanager_StateReplication_InitialSyncFromPeers/RF_=_3 (1.40s) === RUN TestSafeTemplateFilepath === RUN TestSafeTemplateFilepath/should_succeed_if_the_provided_template_is_a_filename === RUN TestSafeTemplateFilepath/template_name_starting_with_/ === RUN TestSafeTemplateFilepath/root_dir === RUN TestSafeTemplateFilepath/root_dir_2 === RUN TestSafeTemplateFilepath/should_fail_if_the_provided_template_is_escaping_the_dir === RUN TestSafeTemplateFilepath/escaping_template_name_that_has_prefix_of_dir_(tmp_is_prefix_of_tmpfile) === RUN TestSafeTemplateFilepath/empty_template_name === RUN TestSafeTemplateFilepath/dot_template_name --- PASS: TestSafeTemplateFilepath (0.01s) --- PASS: TestSafeTemplateFilepath/should_succeed_if_the_provided_template_is_a_filename (0.00s) --- PASS: TestSafeTemplateFilepath/template_name_starting_with_/ (0.00s) --- PASS: TestSafeTemplateFilepath/root_dir (0.00s) --- PASS: TestSafeTemplateFilepath/root_dir_2 (0.00s) --- PASS: TestSafeTemplateFilepath/should_fail_if_the_provided_template_is_escaping_the_dir (0.00s) --- PASS: TestSafeTemplateFilepath/escaping_template_name_that_has_prefix_of_dir_(tmp_is_prefix_of_tmpfile) (0.00s) --- PASS: TestSafeTemplateFilepath/empty_template_name (0.00s) --- PASS: TestSafeTemplateFilepath/dot_template_name (0.00s) === RUN TestStoreTemplateFile --- PASS: TestStoreTemplateFile (0.00s) === RUN TestMultitenantAlertmanager_verifyRateLimitedEmailConfig --- PASS: TestMultitenantAlertmanager_verifyRateLimitedEmailConfig (0.23s) === RUN TestRateLimitedNotifier --- PASS: TestRateLimitedNotifier (1.00s) === RUN TestStatePersister_Position0ShouldWrite --- PASS: TestStatePersister_Position0ShouldWrite (6.11s) === RUN TestStatePersister_Position1ShouldNotWrite --- PASS: TestStatePersister_Position1ShouldNotWrite (5.00s) === RUN TestStateReplication === RUN TestStateReplication/with_a_replication_factor_of_<=_1,_state_is_not_replicated_but_loaded_from_storage. === RUN TestStateReplication/with_a_replication_factor_of_>_1,_state_is_broadcasted_for_replication. --- PASS: TestStateReplication (0.21s) --- PASS: TestStateReplication/with_a_replication_factor_of_<=_1,_state_is_not_replicated_but_loaded_from_storage. (0.11s) --- PASS: TestStateReplication/with_a_replication_factor_of_>_1,_state_is_broadcasted_for_replication. (0.11s) === RUN TestStateReplication_Settle === RUN TestStateReplication_Settle/with_a_replication_factor_of_<=_1,_no_state_can_be_read_from_peers. === RUN TestStateReplication_Settle/with_a_replication_factor_of_>_1,_state_is_read_from_all_peers. === RUN TestStateReplication_Settle/with_full_state_having_no_parts,_nothing_is_merged. === RUN TestStateReplication_Settle/with_an_unknown_key,_parts_in_the_same_state_are_merged. === RUN TestStateReplication_Settle/with_an_unknown_key,_parts_in_other_states_are_merged. === RUN TestStateReplication_Settle/when_reading_from_replicas_fails,_state_is_read_from_storage. === RUN TestStateReplication_Settle/when_reading_from_replicas_and_from_storage_fails,_still_become_ready. === RUN TestStateReplication_Settle/when_user_not_found_in_all_replicas_and_storage,_read_not_counted_as_failure_and_still_become_ready. === RUN TestStateReplication_Settle/when_reading_the_full_state_takes_too_long,_hit_timeout_but_become_ready. --- PASS: TestStateReplication_Settle (1.03s) --- PASS: TestStateReplication_Settle/with_a_replication_factor_of_<=_1,_no_state_can_be_read_from_peers. (0.00s) --- PASS: TestStateReplication_Settle/with_a_replication_factor_of_>_1,_state_is_read_from_all_peers. (0.00s) --- PASS: TestStateReplication_Settle/with_full_state_having_no_parts,_nothing_is_merged. (0.00s) --- PASS: TestStateReplication_Settle/with_an_unknown_key,_parts_in_the_same_state_are_merged. (0.00s) --- PASS: TestStateReplication_Settle/with_an_unknown_key,_parts_in_other_states_are_merged. (0.00s) --- PASS: TestStateReplication_Settle/when_reading_from_replicas_fails,_state_is_read_from_storage. (0.00s) --- PASS: TestStateReplication_Settle/when_reading_from_replicas_and_from_storage_fails,_still_become_ready. (0.00s) --- PASS: TestStateReplication_Settle/when_user_not_found_in_all_replicas_and_storage,_read_not_counted_as_failure_and_still_become_ready. (0.01s) --- PASS: TestStateReplication_Settle/when_reading_the_full_state_takes_too_long,_hit_timeout_but_become_ready. (1.00s) === RUN TestStateReplication_GetFullState === RUN TestStateReplication_GetFullState/no_keys === RUN TestStateReplication_GetFullState/zero_length_data === RUN TestStateReplication_GetFullState/keys_with_data --- PASS: TestStateReplication_GetFullState (0.00s) --- PASS: TestStateReplication_GetFullState/no_keys (0.00s) --- PASS: TestStateReplication_GetFullState/zero_length_data (0.00s) --- PASS: TestStateReplication_GetFullState/keys_with_data (0.00s) === CONT TestIsHealthyForAlertmanagerOperations === RUN TestIsHealthyForAlertmanagerOperations/ACTIVE_instance_with_last_keepalive_newer_than_timeout === RUN TestIsHealthyForAlertmanagerOperations/ACTIVE_instance_with_last_keepalive_older_than_timeout === RUN TestIsHealthyForAlertmanagerOperations/JOINING_instance_with_last_keepalive_newer_than_timeout === RUN TestIsHealthyForAlertmanagerOperations/LEAVING_instance_with_last_keepalive_newer_than_timeout === RUN TestIsHealthyForAlertmanagerOperations/PENDING_instance_with_last_keepalive_newer_than_timeout --- PASS: TestIsHealthyForAlertmanagerOperations (0.00s) --- PASS: TestIsHealthyForAlertmanagerOperations/ACTIVE_instance_with_last_keepalive_newer_than_timeout (0.00s) --- PASS: TestIsHealthyForAlertmanagerOperations/ACTIVE_instance_with_last_keepalive_older_than_timeout (0.00s) --- PASS: TestIsHealthyForAlertmanagerOperations/JOINING_instance_with_last_keepalive_newer_than_timeout (0.00s) --- PASS: TestIsHealthyForAlertmanagerOperations/LEAVING_instance_with_last_keepalive_newer_than_timeout (0.00s) --- PASS: TestIsHealthyForAlertmanagerOperations/PENDING_instance_with_last_keepalive_newer_than_timeout (0.00s) PASS ok github.com/grafana/mimir/pkg/alertmanager 61.075s ? github.com/grafana/mimir/pkg/alertmanager/alertmanagerpb [no test files] ? github.com/grafana/mimir/pkg/alertmanager/alertspb [no test files] === RUN TestAlertStore_ListAllUsers --- PASS: TestAlertStore_ListAllUsers (0.00s) === RUN TestAlertStore_SetAndGetAlertConfig --- PASS: TestAlertStore_SetAndGetAlertConfig (0.00s) === RUN TestStore_GetAlertConfigs --- PASS: TestStore_GetAlertConfigs (0.00s) === RUN TestAlertStore_DeleteAlertConfig --- PASS: TestAlertStore_DeleteAlertConfig (0.00s) === RUN TestBucketAlertStore_GetSetDeleteFullState --- PASS: TestBucketAlertStore_GetSetDeleteFullState (0.00s) PASS ok github.com/grafana/mimir/pkg/alertmanager/alertstore 0.242s testing: warning: no tests to run PASS ok github.com/grafana/mimir/pkg/alertmanager/alertstore/bucketclient 0.185s [no tests to run] === RUN TestStore_ListAllUsers --- PASS: TestStore_ListAllUsers (0.01s) === RUN TestStore_GetAlertConfig --- PASS: TestStore_GetAlertConfig (0.02s) === RUN TestStore_GetAlertConfigs --- PASS: TestStore_GetAlertConfigs (0.01s) === RUN TestStore_FullState --- PASS: TestStore_FullState (0.00s) PASS ok github.com/grafana/mimir/pkg/alertmanager/alertstore/local 0.204s === RUN TestV1Alerts --- PASS: TestV1Alerts (0.00s) === RUN TestMergeV1Alerts === RUN TestMergeV1Alerts/no_alerts,_should_return_an_empty_list === RUN TestMergeV1Alerts/one_alert,_should_return_the_alert === RUN TestMergeV1Alerts/two_alerts,_should_return_two_alerts === RUN TestMergeV1Alerts/three_alerts,_should_return_three_alerts === RUN TestMergeV1Alerts/three_alerts_out_of_order,_should_return_three_alerts_in_fingerprint_order === RUN TestMergeV1Alerts/two_identical_alerts,_should_return_one_alert === RUN TestMergeV1Alerts/two_identical_alerts_plus_another,_should_return_two_alerts === RUN TestMergeV1Alerts/two_duplicates_out_of_sync_alerts,_should_return_first_seen_alert === RUN TestMergeV1Alerts/two_duplicates_plus_others,_should_return_first_seen_alert_and_others --- PASS: TestMergeV1Alerts (0.00s) --- PASS: TestMergeV1Alerts/no_alerts,_should_return_an_empty_list (0.00s) --- PASS: TestMergeV1Alerts/one_alert,_should_return_the_alert (0.00s) --- PASS: TestMergeV1Alerts/two_alerts,_should_return_two_alerts (0.00s) --- PASS: TestMergeV1Alerts/three_alerts,_should_return_three_alerts (0.00s) --- PASS: TestMergeV1Alerts/three_alerts_out_of_order,_should_return_three_alerts_in_fingerprint_order (0.00s) --- PASS: TestMergeV1Alerts/two_identical_alerts,_should_return_one_alert (0.00s) --- PASS: TestMergeV1Alerts/two_identical_alerts_plus_another,_should_return_two_alerts (0.00s) --- PASS: TestMergeV1Alerts/two_duplicates_out_of_sync_alerts,_should_return_first_seen_alert (0.00s) --- PASS: TestMergeV1Alerts/two_duplicates_plus_others,_should_return_first_seen_alert_and_others (0.00s) === RUN TestV1SilenceID_ReturnsNewestSilence --- PASS: TestV1SilenceID_ReturnsNewestSilence (0.00s) === RUN TestV1SilenceID_InvalidDifferentIDs --- PASS: TestV1SilenceID_InvalidDifferentIDs (0.00s) === RUN TestV1Silences --- PASS: TestV1Silences (0.00s) === RUN TestV2AlertGroups --- PASS: TestV2AlertGroups (0.00s) === RUN TestMergeV2AlertGroups === RUN TestMergeV2AlertGroups/no_groups,_should_return_no_groups === RUN TestMergeV2AlertGroups/one_group_with_one_alert,_should_return_one_group === RUN TestMergeV2AlertGroups/two_groups_with_different_labels,_should_return_two_groups === RUN TestMergeV2AlertGroups/two_groups_with_different_receiver,_should_return_two_groups === RUN TestMergeV2AlertGroups/two_identical_groups_with_different_alerts,_should_return_one_group_with_two_alerts === RUN TestMergeV2AlertGroups/two_identical_groups_with_identical_alerts,_should_return_one_group_with_one_alert === RUN TestMergeV2AlertGroups/two_identical_groups_with_diverged_alerts,_should_return_one_group_with_the_newer_alert === RUN TestMergeV2AlertGroups/two_sets_of_identical_groups_with_single_alerts,_should_merge_all_into_two_groups_ === RUN TestMergeV2AlertGroups/many_unordered_groups,_should_return_groups_ordered_by_labels_then_receiver --- PASS: TestMergeV2AlertGroups (0.00s) --- PASS: TestMergeV2AlertGroups/no_groups,_should_return_no_groups (0.00s) --- PASS: TestMergeV2AlertGroups/one_group_with_one_alert,_should_return_one_group (0.00s) --- PASS: TestMergeV2AlertGroups/two_groups_with_different_labels,_should_return_two_groups (0.00s) --- PASS: TestMergeV2AlertGroups/two_groups_with_different_receiver,_should_return_two_groups (0.00s) --- PASS: TestMergeV2AlertGroups/two_identical_groups_with_different_alerts,_should_return_one_group_with_two_alerts (0.00s) --- PASS: TestMergeV2AlertGroups/two_identical_groups_with_identical_alerts,_should_return_one_group_with_one_alert (0.00s) --- PASS: TestMergeV2AlertGroups/two_identical_groups_with_diverged_alerts,_should_return_one_group_with_the_newer_alert (0.00s) --- PASS: TestMergeV2AlertGroups/two_sets_of_identical_groups_with_single_alerts,_should_merge_all_into_two_groups_ (0.00s) --- PASS: TestMergeV2AlertGroups/many_unordered_groups,_should_return_groups_ordered_by_labels_then_receiver (0.00s) === RUN TestV2Alerts --- PASS: TestV2Alerts (0.00s) === RUN TestMergeV2Alerts === RUN TestMergeV2Alerts/no_alerts,_should_return_an_empty_list === RUN TestMergeV2Alerts/one_alert,_should_return_the_alert === RUN TestMergeV2Alerts/two_alerts,_should_return_two_alerts === RUN TestMergeV2Alerts/three_alerts,_should_return_three_alerts === RUN TestMergeV2Alerts/three_alerts_out_of_order,_should_return_three_alerts_in_fingerprint_order === RUN TestMergeV2Alerts/two_identical_alerts,_should_return_one_alert === RUN TestMergeV2Alerts/two_identical_alerts_plus_another,_should_return_two_alerts === RUN TestMergeV2Alerts/two_duplicates_out_of_sync_alerts,_should_return_newer_alert === RUN TestMergeV2Alerts/two_duplicates_out_of_sync_alerts_(newer_first),_should_return_newer_alert === RUN TestMergeV2Alerts/two_duplicates_plus_others,_should_return_newer_alert_and_others --- PASS: TestMergeV2Alerts (0.00s) --- PASS: TestMergeV2Alerts/no_alerts,_should_return_an_empty_list (0.00s) --- PASS: TestMergeV2Alerts/one_alert,_should_return_the_alert (0.00s) --- PASS: TestMergeV2Alerts/two_alerts,_should_return_two_alerts (0.00s) --- PASS: TestMergeV2Alerts/three_alerts,_should_return_three_alerts (0.00s) --- PASS: TestMergeV2Alerts/three_alerts_out_of_order,_should_return_three_alerts_in_fingerprint_order (0.00s) --- PASS: TestMergeV2Alerts/two_identical_alerts,_should_return_one_alert (0.00s) --- PASS: TestMergeV2Alerts/two_identical_alerts_plus_another,_should_return_two_alerts (0.00s) --- PASS: TestMergeV2Alerts/two_duplicates_out_of_sync_alerts,_should_return_newer_alert (0.00s) --- PASS: TestMergeV2Alerts/two_duplicates_out_of_sync_alerts_(newer_first),_should_return_newer_alert (0.00s) --- PASS: TestMergeV2Alerts/two_duplicates_plus_others,_should_return_newer_alert_and_others (0.00s) === RUN TestV2SilenceId_ReturnsNewestSilence --- PASS: TestV2SilenceId_ReturnsNewestSilence (0.00s) === RUN TestV2SilenceID_InvalidDifferentIDs --- PASS: TestV2SilenceID_InvalidDifferentIDs (0.00s) === RUN TestV2Silences --- PASS: TestV2Silences (0.00s) === RUN TestMergeV2Silences === RUN TestMergeV2Silences/no_silences,_should_return_an_empty_list === RUN TestMergeV2Silences/one_silence,_should_return_the_silence === RUN TestMergeV2Silences/two_silences,_should_return_two_silences === RUN TestMergeV2Silences/three_silences,_should_return_three_silences === RUN TestMergeV2Silences/three_active_silences_out_of_order,_should_return_three_silences_in_expiry_order === RUN TestMergeV2Silences/two_identical_silences,_should_return_one_silence === RUN TestMergeV2Silences/two_identical_silences_plus_another,_should_return_two_silences === RUN TestMergeV2Silences/two_duplicates_out_of_sync_silences,_should_return_newer_silence === RUN TestMergeV2Silences/two_duplicates_out_of_sync_silences_(newer_first),_should_return_newer_silence === RUN TestMergeV2Silences/two_duplicates_plus_others,_should_return_newer_silence_and_others --- PASS: TestMergeV2Silences (0.00s) --- PASS: TestMergeV2Silences/no_silences,_should_return_an_empty_list (0.00s) --- PASS: TestMergeV2Silences/one_silence,_should_return_the_silence (0.00s) --- PASS: TestMergeV2Silences/two_silences,_should_return_two_silences (0.00s) --- PASS: TestMergeV2Silences/three_silences,_should_return_three_silences (0.00s) --- PASS: TestMergeV2Silences/three_active_silences_out_of_order,_should_return_three_silences_in_expiry_order (0.00s) --- PASS: TestMergeV2Silences/two_identical_silences,_should_return_one_silence (0.00s) --- PASS: TestMergeV2Silences/two_identical_silences_plus_another,_should_return_two_silences (0.00s) --- PASS: TestMergeV2Silences/two_duplicates_out_of_sync_silences,_should_return_newer_silence (0.00s) --- PASS: TestMergeV2Silences/two_duplicates_out_of_sync_silences_(newer_first),_should_return_newer_silence (0.00s) --- PASS: TestMergeV2Silences/two_duplicates_plus_others,_should_return_newer_silence_and_others (0.00s) PASS ok github.com/grafana/mimir/pkg/alertmanager/merger 0.285s === RUN TestNewApiWithoutSourceIPExtractor --- PASS: TestNewApiWithoutSourceIPExtractor (0.00s) === RUN TestNewApiWithSourceIPExtractor --- PASS: TestNewApiWithSourceIPExtractor (0.00s) === RUN TestNewApiWithInvalidSourceIPExtractor --- PASS: TestNewApiWithInvalidSourceIPExtractor (0.00s) === RUN TestApiGzip === RUN TestApiGzip/happy_case_gzip === RUN TestApiGzip/gzip_with_priority_header === RUN TestApiGzip/gzip_because_any_is_accepted === RUN TestApiGzip/no_gzip_because_no_header === RUN TestApiGzip/no_gzip_because_not_accepted === RUN TestApiGzip/no_gzip_because_small_payload === RUN TestApiGzip/forced_gzip_with_small_payload === RUN TestApiGzip/gzip_disabled_endpoint === RUN TestApiGzip/compressed_with_gzip --- PASS: TestApiGzip (0.12s) --- PASS: TestApiGzip/happy_case_gzip (0.02s) --- PASS: TestApiGzip/gzip_with_priority_header (0.03s) --- PASS: TestApiGzip/gzip_because_any_is_accepted (0.03s) --- PASS: TestApiGzip/no_gzip_because_no_header (0.01s) --- PASS: TestApiGzip/no_gzip_because_not_accepted (0.01s) --- PASS: TestApiGzip/no_gzip_because_small_payload (0.00s) --- PASS: TestApiGzip/forced_gzip_with_small_payload (0.01s) --- PASS: TestApiGzip/gzip_disabled_endpoint (0.00s) --- PASS: TestApiGzip/compressed_with_gzip (0.00s) === RUN TestIndexHandlerPrefix --- PASS: TestIndexHandlerPrefix (0.01s) === RUN TestIndexPageContent --- PASS: TestIndexPageContent (0.00s) === RUN TestConfigDiffHandler === RUN TestConfigDiffHandler/no_config_parameters_overridden === RUN TestConfigDiffHandler/slice_changed === RUN TestConfigDiffHandler/string_in_nested_struct_changed === RUN TestConfigDiffHandler/bool_in_nested_struct_changed === RUN TestConfigDiffHandler/test_invalid_input --- PASS: TestConfigDiffHandler (0.01s) --- PASS: TestConfigDiffHandler/no_config_parameters_overridden (0.00s) --- PASS: TestConfigDiffHandler/slice_changed (0.00s) --- PASS: TestConfigDiffHandler/string_in_nested_struct_changed (0.00s) --- PASS: TestConfigDiffHandler/bool_in_nested_struct_changed (0.00s) --- PASS: TestConfigDiffHandler/test_invalid_input (0.00s) === RUN TestConfigOverrideHandler --- PASS: TestConfigOverrideHandler (0.00s) PASS ok github.com/grafana/mimir/pkg/api 0.652s ? github.com/grafana/mimir/pkg/api/error [no test files] === RUN TestCompressionConfig_Validate === RUN TestCompressionConfig_Validate/should_pass_with_default_config === RUN TestCompressionConfig_Validate/should_pass_with_snappy_compression === RUN TestCompressionConfig_Validate/should_fail_with_unsupported_compression --- PASS: TestCompressionConfig_Validate (0.00s) --- PASS: TestCompressionConfig_Validate/should_pass_with_default_config (0.00s) --- PASS: TestCompressionConfig_Validate/should_pass_with_snappy_compression (0.00s) --- PASS: TestCompressionConfig_Validate/should_fail_with_unsupported_compression (0.00s) === RUN TestSnappyCache === RUN TestSnappyCache/Fetch()_should_return_empty_results_if_no_key_has_been_found === RUN TestSnappyCache/Fetch()_should_return_previously_set_keys === RUN TestSnappyCache/Fetch()_should_skip_entries_failing_to_decode --- PASS: TestSnappyCache (0.00s) --- PASS: TestSnappyCache/Fetch()_should_return_empty_results_if_no_key_has_been_found (0.00s) --- PASS: TestSnappyCache/Fetch()_should_return_previously_set_keys (0.00s) --- PASS: TestSnappyCache/Fetch()_should_skip_entries_failing_to_decode (0.00s) === RUN TestLRUCache_StoreFetch --- PASS: TestLRUCache_StoreFetch (0.00s) === RUN TestLRUCache_Evictions --- PASS: TestLRUCache_Evictions (0.00s) === RUN TestMemcachedIndexCacheConfig_GetAddresses === RUN TestMemcachedIndexCacheConfig_GetAddresses/one_address === RUN TestMemcachedIndexCacheConfig_GetAddresses/two_addresses === RUN TestMemcachedIndexCacheConfig_GetAddresses/no_addresses --- PASS: TestMemcachedIndexCacheConfig_GetAddresses (0.00s) --- PASS: TestMemcachedIndexCacheConfig_GetAddresses/one_address (0.00s) --- PASS: TestMemcachedIndexCacheConfig_GetAddresses/two_addresses (0.00s) --- PASS: TestMemcachedIndexCacheConfig_GetAddresses/no_addresses (0.00s) === RUN TestVersioned === RUN TestVersioned/happy_case:_can_store_and_retrieve === RUN TestVersioned/different_versions_use_different_datasets --- PASS: TestVersioned (0.00s) --- PASS: TestVersioned/happy_case:_can_store_and_retrieve (0.00s) --- PASS: TestVersioned/different_versions_use_different_datasets (0.00s) PASS ok github.com/grafana/mimir/pkg/cache 0.109s ? github.com/grafana/mimir/pkg/cacheutil [no test files] === RUN TestMultitenantCompactor_HandleBlockUpload_Create === RUN TestMultitenantCompactor_HandleBlockUpload_Create/missing_tenant_ID === RUN TestMultitenantCompactor_HandleBlockUpload_Create/missing_block_ID === RUN TestMultitenantCompactor_HandleBlockUpload_Create/invalid_block_ID === RUN TestMultitenantCompactor_HandleBlockUpload_Create/missing_body block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/malformed_body block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/invalid_file_path block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/missing_file_size block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/invalid_minTime block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/invalid_maxTime block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/maxTime_before_minTime block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/block_before_retention_period block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/invalid_version block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/ignore_retention_period_if_==_0 block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) block_upload_test.go:555: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/ignore_retention_period_if_<_0 block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) block_upload_test.go:555: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/invalid_compactor_shard_ID_label block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/failure_checking_for_complete_block block_upload_test.go:555: PASS: Exists(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/complete_block_already_exists block_upload_test.go:555: PASS: Exists(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/failure_uploading_meta_file block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) block_upload_test.go:555: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/block_upload_disabled === RUN TestMultitenantCompactor_HandleBlockUpload_Create/valid_request block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) block_upload_test.go:555: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_with_empty_compactor_shard_ID_label block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) block_upload_test.go:555: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_without_compactor_shard_ID_label block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) block_upload_test.go:555: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_with_different_block_ID_in_meta_file block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) block_upload_test.go:555: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_when_both_in-flight_meta_file_and_complete_meta_file_exist_in_object_storage === RUN TestMultitenantCompactor_HandleBlockUpload_Create/invalid_request_when_in-flight_meta_file_exists_in_object_storage === RUN TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_when_same_in-flight_meta_file_exists_in_object_storage === RUN TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_when_different_in-flight_meta_file_exists_in_object_storage --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create (0.13s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/missing_tenant_ID (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/missing_block_ID (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/invalid_block_ID (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/missing_body (0.01s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/malformed_body (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/invalid_file_path (0.02s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/missing_file_size (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/invalid_minTime (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/invalid_maxTime (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/maxTime_before_minTime (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/block_before_retention_period (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/invalid_version (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/ignore_retention_period_if_==_0 (0.01s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/ignore_retention_period_if_<_0 (0.01s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/invalid_compactor_shard_ID_label (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/failure_checking_for_complete_block (0.01s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/complete_block_already_exists (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/failure_uploading_meta_file (0.01s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/block_upload_disabled (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/valid_request (0.01s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_with_empty_compactor_shard_ID_label (0.01s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_without_compactor_shard_ID_label (0.01s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_with_different_block_ID_in_meta_file (0.01s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_when_both_in-flight_meta_file_and_complete_meta_file_exist_in_object_storage (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/invalid_request_when_in-flight_meta_file_exists_in_object_storage (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_when_same_in-flight_meta_file_exists_in_object_storage (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_when_different_in-flight_meta_file_exists_in_object_storage (0.00s) === RUN TestMultitenantCompactor_UploadBlockFile === RUN TestMultitenantCompactor_UploadBlockFile/without_tenant_ID === RUN TestMultitenantCompactor_UploadBlockFile/without_block_ID === RUN TestMultitenantCompactor_UploadBlockFile/invalid_block_ID === RUN TestMultitenantCompactor_UploadBlockFile/without_path === RUN TestMultitenantCompactor_UploadBlockFile/invalid_path === RUN TestMultitenantCompactor_UploadBlockFile/empty_file === RUN TestMultitenantCompactor_UploadBlockFile/attempt_block_metadata_file === RUN TestMultitenantCompactor_UploadBlockFile/attempt_in-flight_block_metadata_file === RUN TestMultitenantCompactor_UploadBlockFile/block_upload_disabled === RUN TestMultitenantCompactor_UploadBlockFile/complete_block_already_exists block_upload_test.go:967: PASS: Exists(string,string) === RUN TestMultitenantCompactor_UploadBlockFile/failure_checking_for_complete_block block_upload_test.go:967: PASS: Exists(string,string) === RUN TestMultitenantCompactor_UploadBlockFile/failure_checking_for_in-flight_meta_file block_upload_test.go:967: PASS: Exists(string,string) block_upload_test.go:967: PASS: Get(string,string) === RUN TestMultitenantCompactor_UploadBlockFile/missing_in-flight_meta_file block_upload_test.go:967: PASS: Exists(string,string) block_upload_test.go:967: PASS: Get(string,string) === RUN TestMultitenantCompactor_UploadBlockFile/file_upload_fails block_upload_test.go:967: PASS: Exists(string,string) block_upload_test.go:967: PASS: Get(string,string) block_upload_test.go:967: PASS: Get(string,string) block_upload_test.go:967: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_UploadBlockFile/invalid_file_size block_upload_test.go:967: PASS: Exists(string,string) block_upload_test.go:967: PASS: Get(string,string) block_upload_test.go:967: PASS: Get(string,string) === RUN TestMultitenantCompactor_UploadBlockFile/unexpected_file block_upload_test.go:967: PASS: Exists(string,string) block_upload_test.go:967: PASS: Get(string,string) block_upload_test.go:967: PASS: Get(string,string) === RUN TestMultitenantCompactor_UploadBlockFile/valid_request block_upload_test.go:967: PASS: Exists(string,string) block_upload_test.go:967: PASS: Get(string,string) block_upload_test.go:967: PASS: Get(string,string) block_upload_test.go:967: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_UploadBlockFile/multiple_sequential_uploads_of_same_file --- PASS: TestMultitenantCompactor_UploadBlockFile (0.04s) --- PASS: TestMultitenantCompactor_UploadBlockFile/without_tenant_ID (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/without_block_ID (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/invalid_block_ID (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/without_path (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/invalid_path (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/empty_file (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/attempt_block_metadata_file (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/attempt_in-flight_block_metadata_file (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/block_upload_disabled (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/complete_block_already_exists (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/failure_checking_for_complete_block (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/failure_checking_for_in-flight_meta_file (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/missing_in-flight_meta_file (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/file_upload_fails (0.01s) --- PASS: TestMultitenantCompactor_UploadBlockFile/invalid_file_size (0.01s) --- PASS: TestMultitenantCompactor_UploadBlockFile/unexpected_file (0.01s) --- PASS: TestMultitenantCompactor_UploadBlockFile/valid_request (0.01s) --- PASS: TestMultitenantCompactor_UploadBlockFile/multiple_sequential_uploads_of_same_file (0.00s) === RUN TestMultitenantCompactor_HandleBlockUpload_Complete === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/without_tenant_ID === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/without_block_ID === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/invalid_block_ID === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/block_upload_disabled === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/complete_block_already_exists block_upload_test.go:1264: PASS: Exists(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/checking_for_complete_block_fails block_upload_test.go:1264: PASS: Exists(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/missing_in-flight_meta_file block_upload_test.go:1264: PASS: Exists(string,string) block_upload_test.go:1264: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/downloading_in-flight_meta_file_fails block_upload_test.go:1264: PASS: Exists(string,string) block_upload_test.go:1264: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/corrupt_in-flight_meta_file block_upload_test.go:1264: PASS: Exists(string,string) block_upload_test.go:1264: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/uploading_meta_file_fails block_upload_test.go:1264: PASS: Exists(string,string) block_upload_test.go:1264: PASS: Get(string,string) block_upload_test.go:1264: PASS: Get(string,string) block_upload_test.go:1264: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/removing_in-flight_meta_file_fails block_upload_test.go:1264: PASS: Exists(string,string) block_upload_test.go:1264: PASS: Get(string,string) block_upload_test.go:1264: PASS: Get(string,string) block_upload_test.go:1264: PASS: Upload(string,string,string) block_upload_test.go:1264: PASS: Delete(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/valid_request block_upload_test.go:1264: PASS: Exists(string,string) block_upload_test.go:1264: PASS: Get(string,string) block_upload_test.go:1264: PASS: Get(string,string) block_upload_test.go:1264: PASS: Upload(string,string,string) block_upload_test.go:1264: PASS: Delete(string,string) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete (0.04s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/without_tenant_ID (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/without_block_ID (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/invalid_block_ID (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/block_upload_disabled (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/complete_block_already_exists (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/checking_for_complete_block_fails (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/missing_in-flight_meta_file (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/downloading_in-flight_meta_file_fails (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/corrupt_in-flight_meta_file (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/uploading_meta_file_fails (0.01s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/removing_in-flight_meta_file_fails (0.01s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/valid_request (0.01s) === RUN TestMultitenantCompactor_GetBlockUploadStateHandler === RUN TestMultitenantCompactor_GetBlockUploadStateHandler/block_doesn't_exist === RUN TestMultitenantCompactor_GetBlockUploadStateHandler/complete_block === RUN TestMultitenantCompactor_GetBlockUploadStateHandler/upload_in_progress === RUN TestMultitenantCompactor_GetBlockUploadStateHandler/validating === RUN TestMultitenantCompactor_GetBlockUploadStateHandler/validation_failed === RUN TestMultitenantCompactor_GetBlockUploadStateHandler/stale_validation_file --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler (0.01s) --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler/block_doesn't_exist (0.00s) --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler/complete_block (0.00s) --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler/upload_in_progress (0.00s) --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler/validating (0.00s) --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler/validation_failed (0.00s) --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler/stale_validation_file (0.00s) === RUN TestBlocksCleaner === RUN TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=0s === PAUSE TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=0s === RUN TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=2h0m0s === PAUSE TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=2h0m0s === RUN TestBlocksCleaner/concurrency=2,_tenant_deletion_delay=0s === PAUSE TestBlocksCleaner/concurrency=2,_tenant_deletion_delay=0s === RUN TestBlocksCleaner/concurrency=10,_tenant_deletion_delay=0s === PAUSE TestBlocksCleaner/concurrency=10,_tenant_deletion_delay=0s === CONT TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=0s === CONT TestBlocksCleaner/concurrency=2,_tenant_deletion_delay=0s === CONT TestBlocksCleaner/concurrency=10,_tenant_deletion_delay=0s === CONT TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=2h0m0s --- PASS: TestBlocksCleaner (0.00s) --- PASS: TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=2h0m0s (52.37s) --- PASS: TestBlocksCleaner/concurrency=10,_tenant_deletion_delay=0s (53.19s) --- PASS: TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=0s (53.55s) --- PASS: TestBlocksCleaner/concurrency=2,_tenant_deletion_delay=0s (54.10s) === RUN TestBlocksCleaner_ShouldContinueOnBlockDeletionFailure --- PASS: TestBlocksCleaner_ShouldContinueOnBlockDeletionFailure (19.40s) === RUN TestBlocksCleaner_ShouldRebuildBucketIndexOnCorruptedOne --- PASS: TestBlocksCleaner_ShouldRebuildBucketIndexOnCorruptedOne (14.47s) === RUN TestBlocksCleaner_ShouldRemoveMetricsForTenantsNotBelongingAnymoreToTheShard --- PASS: TestBlocksCleaner_ShouldRemoveMetricsForTenantsNotBelongingAnymoreToTheShard (26.40s) === RUN TestBlocksCleaner_ShouldNotCleanupUserThatDoesntBelongToShardAnymore --- PASS: TestBlocksCleaner_ShouldNotCleanupUserThatDoesntBelongToShardAnymore (11.47s) === RUN TestBlocksCleaner_ListBlocksOutsideRetentionPeriod --- PASS: TestBlocksCleaner_ListBlocksOutsideRetentionPeriod (17.71s) === RUN TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 17.194989ms logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 6.260996ms logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level debug component cleaner user user-1 msg applying retention retention 9h0m0s logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 11.322993ms logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 8.821995ms logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level debug component cleaner user user-1 msg applying retention retention 7h0m0s logger.go:22: level info component cleaner user user-1 msg applied retention: marking block for deletion block 01GQ0FFQ6XHRB6N2EDAV8CDC80 maxTime 1673950947000 logger.go:22: level info component cleaner user user-1 msg block has been marked for deletion block 01GQ0FFQ6XHRB6N2EDAV8CDC80 logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 14.747991ms logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 5.964996ms logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level debug component cleaner user user-1 msg applying retention retention 7h0m0s logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 15.06599ms logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 8.761994ms logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level debug component cleaner user user-1 msg applying retention retention 7h0m0s logger.go:22: level debug component cleaner user user-1 msg deleted file file 01GQ0FFQ6XHRB6N2EDAV8CDC80/meta.json bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod2426905277/001 logger.go:22: level debug component cleaner user user-1 msg deleted file file 01GQ0FFQ6XHRB6N2EDAV8CDC80/chunks/000001 bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod2426905277/001 logger.go:22: level debug component cleaner user user-1 msg deleted file file 01GQ0FFQ6XHRB6N2EDAV8CDC80/index bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod2426905277/001 logger.go:22: level debug component cleaner user user-1 msg deleted file file 01GQ0FFQ6XHRB6N2EDAV8CDC80/tombstones bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod2426905277/001 logger.go:22: level debug component cleaner user user-1 msg deleted file file 01GQ0FFQ6XHRB6N2EDAV8CDC80/deletion-mark.json bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod2426905277/001 logger.go:22: level info component cleaner user user-1 msg deleted block marked for deletion block 01GQ0FFQ6XHRB6N2EDAV8CDC80 logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 17.405989ms logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 12.429993ms logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level debug component cleaner user user-1 msg applying retention retention 7h0m0s logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 12.545992ms logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level debug component cleaner user user-2 msg applying retention retention 5h0m0s logger.go:22: level info component cleaner user user-2 msg applied retention: marking block for deletion block 01GQ0FG232BJ26DEXQKEXQ1S75 maxTime 1673950958000 logger.go:22: level info component cleaner user user-2 msg block has been marked for deletion block 01GQ0FG232BJ26DEXQKEXQ1S75 logger.go:22: level info component cleaner user user-2 msg applied retention: marking block for deletion block 01GQ0FG8YEKWXVHFAQNXHTM5H5 maxTime 1673958165000 logger.go:22: level info component cleaner user user-2 msg block has been marked for deletion block 01GQ0FG8YEKWXVHFAQNXHTM5H5 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GQ0FG232BJ26DEXQKEXQ1S75/meta.json bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod2426905277/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GQ0FG232BJ26DEXQKEXQ1S75/chunks/000001 bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod2426905277/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GQ0FG232BJ26DEXQKEXQ1S75/index bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod2426905277/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GQ0FG232BJ26DEXQKEXQ1S75/tombstones bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod2426905277/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GQ0FG232BJ26DEXQKEXQ1S75/deletion-mark.json bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod2426905277/001 logger.go:22: level info component cleaner user user-2 msg deleted block marked for deletion block 01GQ0FG232BJ26DEXQKEXQ1S75 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GQ0FG8YEKWXVHFAQNXHTM5H5/meta.json bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod2426905277/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GQ0FG8YEKWXVHFAQNXHTM5H5/chunks/000001 bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod2426905277/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GQ0FG8YEKWXVHFAQNXHTM5H5/index bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod2426905277/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GQ0FG8YEKWXVHFAQNXHTM5H5/tombstones bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod2426905277/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GQ0FG8YEKWXVHFAQNXHTM5H5/deletion-mark.json bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod2426905277/001 logger.go:22: level info component cleaner user user-2 msg deleted block marked for deletion block 01GQ0FG8YEKWXVHFAQNXHTM5H5 logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 30.53698ms --- PASS: TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod (24.80s) === RUN TestBlocksCleaner_ShouldRemovePartialBlocksOutsideDelayPeriod logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level warn component cleaner user user-1 msg skipped partial block when updating bucket index block 01GQ0FGFF2965B9A8VV5FDW9TP logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 6.392996ms logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level warn component cleaner user user-1 msg skipped partial block when updating bucket index block 01GQ0FGFF2965B9A8VV5FDW9TP logger.go:22: level info component cleaner user user-1 msg stale partial block found: marking block for deletion block 01GQ0FGFF2965B9A8VV5FDW9TP last modified 2023-01-18 02:22:58.178834601 +0800 CST logger.go:22: level info component cleaner user user-1 msg block has been marked for deletion block 01GQ0FGFF2965B9A8VV5FDW9TP logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 8.841995ms --- PASS: TestBlocksCleaner_ShouldRemovePartialBlocksOutsideDelayPeriod (11.02s) === RUN TestBlocksCleaner_ShouldNotRemovePartialBlocksInsideDelayPeriod logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level warn component cleaner user user-1 msg skipped partial block when updating bucket index block 01GQ0FGT6ZK7J3P2K3KFBRSY9G logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 13.436991ms logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level error component cleaner user user-2 msg skipped block with corrupted meta.json when updating bucket index block 01GQ0FGZVJR0WE01HB8MTCKMJQ err unmarshal block meta file 01GQ0FGZVJR0WE01HB8MTCKMJQ/meta.json: invalid character 'c' looking for beginning of value: meta.json corrupted logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 12.244993ms --- PASS: TestBlocksCleaner_ShouldNotRemovePartialBlocksInsideDelayPeriod (11.34s) === RUN TestBlocksCleaner_ShouldNotRemovePartialBlocksIfConfiguredDelayIsInvalid --- PASS: TestBlocksCleaner_ShouldNotRemovePartialBlocksIfConfiguredDelayIsInvalid (5.47s) === RUN TestFindMostRecentModifiedTimeForBlock --- PASS: TestFindMostRecentModifiedTimeForBlock (5.16s) === RUN TestSyncer_GarbageCollect_e2e === PAUSE TestSyncer_GarbageCollect_e2e === RUN TestGroupCompactE2E === PAUSE TestGroupCompactE2E === RUN TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly === PAUSE TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly === RUN TestGroupKey === RUN TestGroupKey/#00 === RUN TestGroupKey/#01 === RUN TestGroupKey/#02 === RUN TestGroupKey/#03 --- PASS: TestGroupKey (0.00s) --- PASS: TestGroupKey/#00 (0.00s) --- PASS: TestGroupKey/#01 (0.00s) --- PASS: TestGroupKey/#02 (0.00s) --- PASS: TestGroupKey/#03 (0.00s) === RUN TestGroupMaxMinTime --- PASS: TestGroupMaxMinTime (0.00s) === RUN TestFilterOwnJobs === RUN TestFilterOwnJobs/should_return_all_planned_jobs_if_the_compactor_instance_owns_all_of_them === RUN TestFilterOwnJobs/should_return_no_jobs_if_the_compactor_instance_owns_none_of_them === RUN TestFilterOwnJobs/should_return_some_jobs_if_the_compactor_instance_owns_some_of_them --- PASS: TestFilterOwnJobs (0.00s) --- PASS: TestFilterOwnJobs/should_return_all_planned_jobs_if_the_compactor_instance_owns_all_of_them (0.00s) --- PASS: TestFilterOwnJobs/should_return_no_jobs_if_the_compactor_instance_owns_none_of_them (0.00s) --- PASS: TestFilterOwnJobs/should_return_some_jobs_if_the_compactor_instance_owns_some_of_them (0.00s) === RUN TestBlockMaxTimeDeltas --- PASS: TestBlockMaxTimeDeltas (0.00s) === RUN TestNoCompactionMarkFilter === RUN TestNoCompactionMarkFilter/filter_with_no_deletion_of_blocks_marked_for_no-compaction === RUN TestNoCompactionMarkFilter/filter_with_deletion_enabled === RUN TestNoCompactionMarkFilter/filter_with_deletion_enabled,_but_canceled_context === RUN TestNoCompactionMarkFilter/filtering_block_with_wrong_marker_version --- PASS: TestNoCompactionMarkFilter (0.01s) --- PASS: TestNoCompactionMarkFilter/filter_with_no_deletion_of_blocks_marked_for_no-compaction (0.00s) --- PASS: TestNoCompactionMarkFilter/filter_with_deletion_enabled (0.00s) --- PASS: TestNoCompactionMarkFilter/filter_with_deletion_enabled,_but_canceled_context (0.00s) --- PASS: TestNoCompactionMarkFilter/filtering_block_with_wrong_marker_version (0.00s) === RUN TestConvertCompactionResultToForEachJobs --- PASS: TestConvertCompactionResultToForEachJobs (0.00s) === RUN TestRingConfig_DefaultConfigToLifecyclerConfig --- PASS: TestRingConfig_DefaultConfigToLifecyclerConfig (0.01s) === RUN TestRingConfig_CustomConfigToLifecyclerConfig --- PASS: TestRingConfig_CustomConfigToLifecyclerConfig (0.00s) === RUN TestConfig_ShouldSupportYamlConfig --- PASS: TestConfig_ShouldSupportYamlConfig (0.00s) === RUN TestConfig_ShouldSupportCliFlags --- PASS: TestConfig_ShouldSupportCliFlags (0.00s) === RUN TestConfig_Validate === RUN TestConfig_Validate/should_pass_with_only_1_block_range_period === RUN TestConfig_Validate/should_fail_with_non_divisible_block_range_periods === RUN TestConfig_Validate/should_fail_on_unknown_compaction_jobs_order === RUN TestConfig_Validate/should_fail_on_invalid_value_of_max-opening-blocks-concurrency === RUN TestConfig_Validate/should_fail_on_invalid_value_of_max-closing-blocks-concurrency === RUN TestConfig_Validate/should_fail_on_invalid_value_of_symbols-flushers-concurrency === RUN TestConfig_Validate/should_pass_with_the_default_config --- PASS: TestConfig_Validate (0.01s) --- PASS: TestConfig_Validate/should_pass_with_only_1_block_range_period (0.00s) --- PASS: TestConfig_Validate/should_fail_with_non_divisible_block_range_periods (0.00s) --- PASS: TestConfig_Validate/should_fail_on_unknown_compaction_jobs_order (0.00s) --- PASS: TestConfig_Validate/should_fail_on_invalid_value_of_max-opening-blocks-concurrency (0.00s) --- PASS: TestConfig_Validate/should_fail_on_invalid_value_of_max-closing-blocks-concurrency (0.00s) --- PASS: TestConfig_Validate/should_fail_on_invalid_value_of_symbols-flushers-concurrency (0.00s) --- PASS: TestConfig_Validate/should_pass_with_the_default_config (0.00s) === RUN TestMultitenantCompactor_ShouldDoNothingOnNoUserBlocks === PAUSE TestMultitenantCompactor_ShouldDoNothingOnNoUserBlocks === RUN TestMultitenantCompactor_ShouldRetryCompactionOnFailureWhileDiscoveringUsersFromBucket === PAUSE TestMultitenantCompactor_ShouldRetryCompactionOnFailureWhileDiscoveringUsersFromBucket === RUN TestMultitenantCompactor_ShouldIncrementCompactionErrorIfFailedToCompactASingleTenant === PAUSE TestMultitenantCompactor_ShouldIncrementCompactionErrorIfFailedToCompactASingleTenant === RUN TestMultitenantCompactor_ShouldIterateOverUsersAndRunCompaction === PAUSE TestMultitenantCompactor_ShouldIterateOverUsersAndRunCompaction === RUN TestMultitenantCompactor_ShouldStopCompactingTenantOnReachingMaxCompactionTime === PAUSE TestMultitenantCompactor_ShouldStopCompactingTenantOnReachingMaxCompactionTime === RUN TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForDeletion === PAUSE TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForDeletion === RUN TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForNoCompaction === PAUSE TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForNoCompaction === RUN TestMultitenantCompactor_ShouldNotCompactBlocksForUsersMarkedForDeletion === PAUSE TestMultitenantCompactor_ShouldNotCompactBlocksForUsersMarkedForDeletion === RUN TestMultitenantCompactor_ShouldCompactAllUsersOnShardingEnabledButOnlyOneInstanceRunning === PAUSE TestMultitenantCompactor_ShouldCompactAllUsersOnShardingEnabledButOnlyOneInstanceRunning === RUN TestMultitenantCompactor_ShouldCompactOnlyUsersOwnedByTheInstanceOnShardingEnabledAndMultipleInstancesRunning === PAUSE TestMultitenantCompactor_ShouldCompactOnlyUsersOwnedByTheInstanceOnShardingEnabledAndMultipleInstancesRunning === RUN TestMultitenantCompactor_ShouldSkipCompactionForJobsNoMoreOwnedAfterPlanning === PAUSE TestMultitenantCompactor_ShouldSkipCompactionForJobsNoMoreOwnedAfterPlanning === RUN TestMultitenantCompactor_DeleteLocalSyncFiles --- PASS: TestMultitenantCompactor_DeleteLocalSyncFiles (6.58s) === RUN TestMultitenantCompactor_ShouldFailCompactionOnTimeout === PAUSE TestMultitenantCompactor_ShouldFailCompactionOnTimeout === RUN TestOwnUser === RUN TestOwnUser/5_compactors,_sharding_enabled,_no_compactor_shard_size === PAUSE TestOwnUser/5_compactors,_sharding_enabled,_no_compactor_shard_size === RUN TestOwnUser/10_compactors,_sharding_enabled,_with_non-zero_shard_sizes === PAUSE TestOwnUser/10_compactors,_sharding_enabled,_with_non-zero_shard_sizes === RUN TestOwnUser/10_compactors,_sharding_enabled,_with_zero_shard_size === PAUSE TestOwnUser/10_compactors,_sharding_enabled,_with_zero_shard_size === CONT TestOwnUser/5_compactors,_sharding_enabled,_no_compactor_shard_size === CONT TestOwnUser/10_compactors,_sharding_enabled,_with_zero_shard_size === CONT TestOwnUser/10_compactors,_sharding_enabled,_with_non-zero_shard_sizes --- PASS: TestOwnUser (0.02s) --- PASS: TestOwnUser/10_compactors,_sharding_enabled,_with_zero_shard_size (4.39s) --- PASS: TestOwnUser/5_compactors,_sharding_enabled,_no_compactor_shard_size (4.53s) --- PASS: TestOwnUser/10_compactors,_sharding_enabled,_with_non-zero_shard_sizes (5.31s) === RUN TestMultitenantCompactor_OutOfOrderCompaction --- PASS: TestMultitenantCompactor_OutOfOrderCompaction (7.99s) === RUN TestSortJobsBySmallestRangeOldestBlocksFirst === RUN TestSortJobsBySmallestRangeOldestBlocksFirst/split_jobs_are_always_sorted_first === RUN TestSortJobsBySmallestRangeOldestBlocksFirst/should_do_nothing_on_empty_input === RUN TestSortJobsBySmallestRangeOldestBlocksFirst/should_sort_jobs_by_smallest_range,_oldest_blocks_first --- PASS: TestSortJobsBySmallestRangeOldestBlocksFirst (0.01s) --- PASS: TestSortJobsBySmallestRangeOldestBlocksFirst/split_jobs_are_always_sorted_first (0.00s) --- PASS: TestSortJobsBySmallestRangeOldestBlocksFirst/should_do_nothing_on_empty_input (0.00s) --- PASS: TestSortJobsBySmallestRangeOldestBlocksFirst/should_sort_jobs_by_smallest_range,_oldest_blocks_first (0.00s) === RUN TestSortJobsByNewestBlocksFirst === RUN TestSortJobsByNewestBlocksFirst/should_give_precedence_to_smaller_time_ranges_in_case_of_multiple_jobs_with_the_same_max_time job_sorting_test.go:127: sorted jobs: job_sorting_test.go:129: - (minTime: 30 maxTime: 40) job_sorting_test.go:129: - (minTime: 10 maxTime: 40) === RUN TestSortJobsByNewestBlocksFirst/should_give_precedence_to_newest_blocks_over_smaller_time_ranges job_sorting_test.go:127: sorted jobs: job_sorting_test.go:129: - (minTime: 10 maxTime: 40) job_sorting_test.go:129: - (minTime: 10 maxTime: 30) job_sorting_test.go:129: - (minTime: 10 maxTime: 20) === RUN TestSortJobsByNewestBlocksFirst/should_do_nothing_on_empty_input job_sorting_test.go:127: sorted jobs: === RUN TestSortJobsByNewestBlocksFirst/should_sort_jobs_by_newest_blocks_first job_sorting_test.go:127: sorted jobs: job_sorting_test.go:129: - (minTime: 40 maxTime: 80) job_sorting_test.go:129: - (minTime: 10 maxTime: 30) job_sorting_test.go:129: - (minTime: 10 maxTime: 20) --- PASS: TestSortJobsByNewestBlocksFirst (0.00s) --- PASS: TestSortJobsByNewestBlocksFirst/should_give_precedence_to_smaller_time_ranges_in_case_of_multiple_jobs_with_the_same_max_time (0.00s) --- PASS: TestSortJobsByNewestBlocksFirst/should_give_precedence_to_newest_blocks_over_smaller_time_ranges (0.00s) --- PASS: TestSortJobsByNewestBlocksFirst/should_do_nothing_on_empty_input (0.00s) --- PASS: TestSortJobsByNewestBlocksFirst/should_sort_jobs_by_newest_blocks_first (0.00s) === RUN TestLabelRemoverFilter === RUN TestLabelRemoverFilter/should_remove_configured_labels === RUN TestLabelRemoverFilter/should_remove_configured_labels_2 --- PASS: TestLabelRemoverFilter (0.00s) --- PASS: TestLabelRemoverFilter/should_remove_configured_labels (0.00s) --- PASS: TestLabelRemoverFilter/should_remove_configured_labels_2 (0.00s) === RUN TestShardAwareDeduplicateFilter_Filter === RUN TestShardAwareDeduplicateFilter_Filter/compacted_blocks_of_level_3_with_overlapping_sources_of_equal_length === RUN TestShardAwareDeduplicateFilter_Filter/compacted_blocks_of_level_3_with_overlapping_sources_of_different_length_and_different_resolutions === RUN TestShardAwareDeduplicateFilter_Filter/two_blocks_merged_and_split,_with_single_shard === RUN TestShardAwareDeduplicateFilter_Filter/two_compacted_blocks_with_overlapping_sources === RUN TestShardAwareDeduplicateFilter_Filter/3_compacted_blocks_of_level_2_and_one_compacted_block_of_level_3_in_bucket === RUN TestShardAwareDeduplicateFilter_Filter/blocks_with_same_sources_and_different_resolutions === RUN TestShardAwareDeduplicateFilter_Filter/when_invalid_shard_IDs_present,_no_deduplication_happens_for_source_blocks === RUN TestShardAwareDeduplicateFilter_Filter/four_base_blocks_merged_and_split_into_2_separate_shards,_plus_another_level === RUN TestShardAwareDeduplicateFilter_Filter/3_non_compacted_blocks_in_bucket === RUN TestShardAwareDeduplicateFilter_Filter/compacted_block_without_sources_in_bucket === RUN TestShardAwareDeduplicateFilter_Filter/two_compacted_blocks_with_same_sources === RUN TestShardAwareDeduplicateFilter_Filter/four_base_blocks_merged_and_split_into_2_separate_shards === RUN TestShardAwareDeduplicateFilter_Filter/compacted_blocks_of_level_3_with_overlapping_sources_of_different_length === RUN TestShardAwareDeduplicateFilter_Filter/block_with_invalid_shardID_cannot_'include'_its_source_blocks === RUN TestShardAwareDeduplicateFilter_Filter/two_blocks_merged_and_split_into_two,_one_shard_missing === RUN TestShardAwareDeduplicateFilter_Filter/two_blocks_merged_and_split,_with_two_shards === RUN TestShardAwareDeduplicateFilter_Filter/four_base_blocks_merged_and_split_into_2_separate_shards,_plus_another_level,_with_various_resolutions === RUN TestShardAwareDeduplicateFilter_Filter/4_non_compacted_blocks_and_compacted_block_of_level_2_in_bucket === RUN TestShardAwareDeduplicateFilter_Filter/compacted_blocks_with_overlapping_sources === RUN TestShardAwareDeduplicateFilter_Filter/compacted_blocks_with_overlapping_sources_and_different_resolutions --- PASS: TestShardAwareDeduplicateFilter_Filter (0.02s) --- PASS: TestShardAwareDeduplicateFilter_Filter/compacted_blocks_of_level_3_with_overlapping_sources_of_equal_length (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/compacted_blocks_of_level_3_with_overlapping_sources_of_different_length_and_different_resolutions (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/two_blocks_merged_and_split,_with_single_shard (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/two_compacted_blocks_with_overlapping_sources (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/3_compacted_blocks_of_level_2_and_one_compacted_block_of_level_3_in_bucket (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/blocks_with_same_sources_and_different_resolutions (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/when_invalid_shard_IDs_present,_no_deduplication_happens_for_source_blocks (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/four_base_blocks_merged_and_split_into_2_separate_shards,_plus_another_level (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/3_non_compacted_blocks_in_bucket (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/compacted_block_without_sources_in_bucket (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/two_compacted_blocks_with_same_sources (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/four_base_blocks_merged_and_split_into_2_separate_shards (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/compacted_blocks_of_level_3_with_overlapping_sources_of_different_length (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/block_with_invalid_shardID_cannot_'include'_its_source_blocks (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/two_blocks_merged_and_split_into_two,_one_shard_missing (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/two_blocks_merged_and_split,_with_two_shards (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/four_base_blocks_merged_and_split_into_2_separate_shards,_plus_another_level,_with_various_resolutions (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/4_non_compacted_blocks_and_compacted_block_of_level_2_in_bucket (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/compacted_blocks_with_overlapping_sources (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/compacted_blocks_with_overlapping_sources_and_different_resolutions (0.00s) === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/should_correctly_handle_empty_blocks_generated_in_the_splitting_stage level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="not loading tokens from file, tokens file path is empty" level=info component=compactor msg="instance not found in ring, adding with no tokens" ring=compactor level=debug component=compactor msg="JoinAfter expired" ring=compactor level=info component=compactor msg="auto-joining cluster after timeout" ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1673979829 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=8.421995ms duration_ms=8 cached=1 returned=1 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job="stage: split, range start: 7200000, range end: 14400000, shard: 1_of_1, blocks: 01GQ0FJ368FABSYY1VGZ547CTK (min time: 1970-01-01 03:59:59.999 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=1 plan="[01GQ0FJ368FABSYY1VGZ547CTK (min time: 14399999, max time: 14400000)]" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=9.509994ms level=info component=cleaner run_id=1673979829 msg="successfully completed blocks cleanup and maintenance" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GQ0FJ368FABSYY1VGZ547CTK/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=1 plan=[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorshould_correctly_handle_empty_blocks_generated_in_the_splitting_stage3410742052/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GQ0FJ368FABSYY1VGZ547CTK] duration=5.197997ms duration_ms=5 level=info component=compactor msg="compact blocks" count=1 mint=14399999 maxt=14400000 ulid=01GQ0FJ6SM6091ABCMRXHF44ZN sources=[01GQ0FJ368FABSYY1VGZ547CTK] duration=4.884904903s shard=1_of_2 level=info component=compactor msg="compact blocks resulted in empty block" count=1 sources=[01GQ0FJ368FABSYY1VGZ547CTK] duration=4.885164903s shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new="[01GQ0FJ6SM6091ABCMRXHF44ZN 00000000000000000000000000]" blocks=[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorshould_correctly_handle_empty_blocks_generated_in_the_splitting_stage3410742052/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GQ0FJ368FABSYY1VGZ547CTK] duration=4.885369903s duration_ms=4885 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction produced an empty block" shard_id=2_of_2 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorshould_correctly_handle_empty_blocks_generated_in_the_splitting_stage3410742052/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GQ0FJ6SM6091ABCMRXHF44ZN/chunks/000001 dst=01GQ0FJ6SM6091ABCMRXHF44ZN/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorshould_correctly_handle_empty_blocks_generated_in_the_splitting_stage3410742052/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorshould_correctly_handle_empty_blocks_generated_in_the_splitting_stage3410742052/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GQ0FJ6SM6091ABCMRXHF44ZN/index dst=01GQ0FJ6SM6091ABCMRXHF44ZN/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorshould_correctly_handle_empty_blocks_generated_in_the_splitting_stage3410742052/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GQ0FJ6SM6091ABCMRXHF44ZN duration=2.965998ms duration_ms=2 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=4.955997ms duration_ms=4 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GQ0FJ368FABSYY1VGZ547CTK level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GQ0FJ368FABSYY1VGZ547CTK level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=4.898473895s duration_ms=4898 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=3.806998ms duration_ms=3 cached=2 returned=1 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=4.517997ms duration_ms=4 cached=2 returned=1 partial=0 level=info component=compactor msg="lifecycler loop() exited gracefully" ring=compactor level=info component=compactor msg="changing instance state from" old_state=ACTIVE new_state=LEAVING ring=compactor level=info component=compactor msg="lifecycler entering final sleep before shutdown" final_sleep=0s level=debug component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the KV store" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="not loading tokens from file, tokens file path is empty" level=info component=compactor msg="instance not found in ring, adding with no tokens" ring=compactor level=debug component=compactor msg="JoinAfter expired" ring=compactor level=info component=compactor msg="auto-joining cluster after timeout" ring=compactor level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1673979841 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=6.365996ms duration_ms=6 cached=2 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job="stage: split, range start: 7200000, range end: 14400000, shard: 1_of_1, blocks: 01GQ0FJBT5YYD9GGZ7EKAQFPPS (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC),01GQ0FJF658DEAM0T88JC0XBPD (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=5.945997ms level=info component=cleaner run_id=1673979841 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GQ0FJBT5YYD9GGZ7EKAQFPPS (min time: 7200000, max time: 14400000) 01GQ0FJF658DEAM0T88JC0XBPD (min time: 7200000, max time: 14400000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GQ0FJF658DEAM0T88JC0XBPD/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GQ0FJBT5YYD9GGZ7EKAQFPPS/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split593417848/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GQ0FJBT5YYD9GGZ7EKAQFPPS /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split593417848/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GQ0FJF658DEAM0T88JC0XBPD]" duration=8.918995ms duration_ms=8 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01GQ0FJK1900EH0MG1JVD5NHWF sources="[01GQ0FJBT5YYD9GGZ7EKAQFPPS 01GQ0FJF658DEAM0T88JC0XBPD]" duration=6.306115003s shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01GQ0FJK19WNTQTT4GVJ4T1RX7 sources="[01GQ0FJBT5YYD9GGZ7EKAQFPPS 01GQ0FJF658DEAM0T88JC0XBPD]" duration=6.306655003s shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new="[01GQ0FJK1900EH0MG1JVD5NHWF 01GQ0FJK19WNTQTT4GVJ4T1RX7]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split593417848/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GQ0FJBT5YYD9GGZ7EKAQFPPS /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split593417848/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GQ0FJF658DEAM0T88JC0XBPD]" duration=6.307082003s duration_ms=6307 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split593417848/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GQ0FJK19WNTQTT4GVJ4T1RX7/chunks/000001 dst=01GQ0FJK19WNTQTT4GVJ4T1RX7/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split593417848/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split593417848/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GQ0FJK19WNTQTT4GVJ4T1RX7/index dst=01GQ0FJK19WNTQTT4GVJ4T1RX7/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split593417848/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GQ0FJK19WNTQTT4GVJ4T1RX7 duration=4.626997ms duration_ms=4 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split593417848/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GQ0FJK1900EH0MG1JVD5NHWF/chunks/000001 dst=01GQ0FJK1900EH0MG1JVD5NHWF/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split593417848/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split593417848/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GQ0FJK1900EH0MG1JVD5NHWF/index dst=01GQ0FJK1900EH0MG1JVD5NHWF/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split593417848/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GQ0FJK1900EH0MG1JVD5NHWF duration=4.691997ms duration_ms=4 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=15.370991ms duration_ms=15 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GQ0FJBT5YYD9GGZ7EKAQFPPS level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GQ0FJBT5YYD9GGZ7EKAQFPPS level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GQ0FJF658DEAM0T88JC0XBPD level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GQ0FJF658DEAM0T88JC0XBPD level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=6.337163983s duration_ms=6337 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=19.788987ms duration_ms=19 cached=4 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=7.876995ms duration_ms=7 cached=4 returned=2 partial=0 level=info component=compactor msg="lifecycler loop() exited gracefully" ring=compactor level=info component=compactor msg="changing instance state from" old_state=ACTIVE new_state=LEAVING ring=compactor level=info component=compactor msg="lifecycler entering final sleep before shutdown" final_sleep=0s level=debug component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the KV store" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/non-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="not loading tokens from file, tokens file path is empty" level=info component=compactor msg="instance not found in ring, adding with no tokens" ring=compactor level=debug component=compactor msg="JoinAfter expired" ring=compactor level=info component=compactor msg="auto-joining cluster after timeout" ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1673979864 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=8.459994ms duration_ms=8 cached=3 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job="stage: split, range start: 0, range end: 7200000, shard: 1_of_1, blocks: 01GQ0FJSDQNTQ6BZWFSEWMM973 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01GQ0FJZH3X340RBA0SVXWW762 (min time: 1970-01-01 00:07:00 +0000 UTC, max time: 1970-01-01 00:10:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GQ0FJSDQNTQ6BZWFSEWMM973 (min time: 0, max time: 300000) 01GQ0FJZH3X340RBA0SVXWW762 (min time: 420000, max time: 600000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GQ0FJZH3X340RBA0SVXWW762/meta.json level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=22.518986ms level=info component=cleaner run_id=1673979864 msg="successfully completed blocks cleanup and maintenance" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GQ0FJSDQNTQ6BZWFSEWMM973/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split4241447775/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FJSDQNTQ6BZWFSEWMM973 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split4241447775/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FJZH3X340RBA0SVXWW762]" duration=23.574985ms duration_ms=23 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=600000 ulid=01GQ0FK9HEHSRGQGV2R5A3QBC4 sources="[01GQ0FJZH3X340RBA0SVXWW762 01GQ0FJSDQNTQ6BZWFSEWMM973]" duration=10.341519445s shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=600000 ulid=01GQ0FK9HEBX8FCPCYGRKEK837 sources="[01GQ0FJZH3X340RBA0SVXWW762 01GQ0FJSDQNTQ6BZWFSEWMM973]" duration=10.341734445s shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compacted blocks" new="[01GQ0FK9HEHSRGQGV2R5A3QBC4 01GQ0FK9HEBX8FCPCYGRKEK837]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split4241447775/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FJSDQNTQ6BZWFSEWMM973 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split4241447775/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FJZH3X340RBA0SVXWW762]" duration=10.341996445s duration_ms=10341 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split4241447775/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FK9HEBX8FCPCYGRKEK837/chunks/000001 dst=01GQ0FK9HEBX8FCPCYGRKEK837/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split4241447775/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split4241447775/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FK9HEBX8FCPCYGRKEK837/index dst=01GQ0FK9HEBX8FCPCYGRKEK837/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split4241447775/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded block" result_block=01GQ0FK9HEBX8FCPCYGRKEK837 duration=3.576998ms duration_ms=3 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split4241447775/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FK9HEHSRGQGV2R5A3QBC4/chunks/000001 dst=01GQ0FK9HEHSRGQGV2R5A3QBC4/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split4241447775/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split4241447775/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FK9HEHSRGQGV2R5A3QBC4/index dst=01GQ0FK9HEHSRGQGV2R5A3QBC4/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split4241447775/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded block" result_block=01GQ0FK9HEHSRGQGV2R5A3QBC4 duration=10.430994ms duration_ms=10 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=18.750988ms duration_ms=18 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GQ0FJSDQNTQ6BZWFSEWMM973 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01GQ0FJSDQNTQ6BZWFSEWMM973 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GQ0FJZH3X340RBA0SVXWW762 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01GQ0FJZH3X340RBA0SVXWW762 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction job succeeded" duration=10.39639341s duration_ms=10396 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=7.224996ms duration_ms=7 cached=5 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=34.592978ms duration_ms=34 cached=5 returned=3 partial=0 level=info component=compactor msg="lifecycler loop() exited gracefully" ring=compactor level=info component=compactor msg="changing instance state from" old_state=ACTIVE new_state=LEAVING ring=compactor level=info component=compactor msg="lifecycler entering final sleep before shutdown" final_sleep=0s level=debug component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the KV store" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/smaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges level=info component=compactor msg="not loading tokens from file, tokens file path is empty" level=info component=compactor msg="instance not found in ring, adding with no tokens" ring=compactor level=debug component=compactor msg="JoinAfter expired" ring=compactor level=info component=compactor msg="auto-joining cluster after timeout" ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1673979936 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=22.091986ms duration_ms=22 cached=10 returned=10 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=14.586991ms level=info component=cleaner run_id=1673979936 msg="successfully completed blocks cleanup and maintenance" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job="stage: merge, range start: 14400000, range end: 28800000, shard: 1_of_2, blocks: 01GQ0FMB68S4K3BK4PN5AZBKE0 (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 06:00:00 +0000 UTC),01GQ0FMQC4MQFWYXXHGT3KRDJ8 (min time: 1970-01-01 06:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job="stage: merge, range start: 14400000, range end: 28800000, shard: 2_of_2, blocks: 01GQ0FMH2NYVM18MMA8CS2MXW7 (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 06:00:00 +0000 UTC),01GQ0FMX5RQ08NRVETBEDN8HR9 (min time: 1970-01-01 06:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 job="stage: split, range start: 28800000, range end: 36000000, shard: 1_of_1, blocks: 01GQ0FN3MQN7Z43CW12Z6AVGNC (min time: 1970-01-01 08:00:00 +0000 UTC, max time: 1970-01-01 10:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 job="stage: split, range start: 50400000, range end: 57600000, shard: 1_of_1, blocks: 01GQ0FN9QPDQZ7CPBT63FM59Y9 (min time: 1970-01-01 14:00:00 +0000 UTC, max time: 1970-01-01 16:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job="stage: split, range start: 7200000, range end: 14400000, shard: 1_of_1, blocks: 01GQ0FKZDKCQ2RHPY9FNKSRV09 (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC),01GQ0FM57BK0R6D3W5AEQVHB89 (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GQ0FM57BK0R6D3W5AEQVHB89 (min time: 7200000, max time: 14400000) 01GQ0FKZDKCQ2RHPY9FNKSRV09 (min time: 7200000, max time: 14400000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GQ0FM57BK0R6D3W5AEQVHB89/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GQ0FKZDKCQ2RHPY9FNKSRV09/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1298268531/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GQ0FM57BK0R6D3W5AEQVHB89 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1298268531/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GQ0FKZDKCQ2RHPY9FNKSRV09]" duration=11.431993ms duration_ms=11 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01GQ0FNG66BPHFKH0C68ZSD0P8 sources="[01GQ0FM57BK0R6D3W5AEQVHB89 01GQ0FKZDKCQ2RHPY9FNKSRV09]" duration=11.007561022s shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01GQ0FNG66AFEDHJMM9H3XSYQ5 sources="[01GQ0FM57BK0R6D3W5AEQVHB89 01GQ0FKZDKCQ2RHPY9FNKSRV09]" duration=11.007973022s shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new="[01GQ0FNG66BPHFKH0C68ZSD0P8 01GQ0FNG66AFEDHJMM9H3XSYQ5]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1298268531/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GQ0FM57BK0R6D3W5AEQVHB89 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1298268531/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GQ0FKZDKCQ2RHPY9FNKSRV09]" duration=11.008363022s duration_ms=11008 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1298268531/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GQ0FNG66AFEDHJMM9H3XSYQ5/chunks/000001 dst=01GQ0FNG66AFEDHJMM9H3XSYQ5/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1298268531/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1298268531/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GQ0FNG66AFEDHJMM9H3XSYQ5/index dst=01GQ0FNG66AFEDHJMM9H3XSYQ5/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1298268531/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GQ0FNG66AFEDHJMM9H3XSYQ5 duration=4.225997ms duration_ms=4 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1298268531/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GQ0FNG66BPHFKH0C68ZSD0P8/chunks/000001 dst=01GQ0FNG66BPHFKH0C68ZSD0P8/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1298268531/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1298268531/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GQ0FNG66BPHFKH0C68ZSD0P8/index dst=01GQ0FNG66BPHFKH0C68ZSD0P8/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1298268531/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GQ0FNG66BPHFKH0C68ZSD0P8 duration=20.795986ms duration_ms=20 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=27.235982ms duration_ms=27 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GQ0FM57BK0R6D3W5AEQVHB89 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GQ0FM57BK0R6D3W5AEQVHB89 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GQ0FKZDKCQ2RHPY9FNKSRV09 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GQ0FKZDKCQ2RHPY9FNKSRV09 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=11.076889978s duration_ms=11076 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=1 plan="[01GQ0FN3MQN7Z43CW12Z6AVGNC (min time: 28800000, max time: 36000000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GQ0FN3MQN7Z43CW12Z6AVGNC/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=1 plan=[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1298268531/001/compact/0@17241709254077376921-split-1_of_1-28800000-36000000/01GQ0FN3MQN7Z43CW12Z6AVGNC] duration=11.063993ms duration_ms=11 split_merge_compactor_test.go:640: expected , got Diff: --- metric output does not match expectation; want +++ got: @@ -2,3 +2,3 @@ # TYPE cortex_compactor_runs_completed_total counter -cortex_compactor_runs_completed_total 1 +cortex_compactor_runs_completed_total 0 level=info component=compactor msg="lifecycler loop() exited gracefully" ring=compactor level=info component=compactor msg="changing instance state from" old_state=ACTIVE new_state=LEAVING ring=compactor level=info component=compactor msg="lifecycler entering final sleep before shutdown" final_sleep=0s level=debug component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the KV store" ring=compactor level=error component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="compaction job failed" duration=10.259416497s duration_ms=10259 err="compact blocks [/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1298268531/001/compact/0@17241709254077376921-split-1_of_1-28800000-36000000/01GQ0FN3MQN7Z43CW12Z6AVGNC]: 3 errors: populate block: writing block: add series: context canceled; context canceled; context canceled" level=error component=compactor msg="failed to compact user blocks" user=user-1 err="compaction: group 0@17241709254077376921-split-1_of_1-28800000-36000000: compact blocks [/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1298268531/001/compact/0@17241709254077376921-split-1_of_1-28800000-36000000/01GQ0FN3MQN7Z43CW12Z6AVGNC]: 3 errors: populate block: writing block: add series: context canceled; context canceled; context canceled" === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="not loading tokens from file, tokens file path is empty" level=info component=compactor msg="instance not found in ring, adding with no tokens" ring=compactor level=debug component=compactor msg="JoinAfter expired" ring=compactor level=info component=compactor msg="auto-joining cluster after timeout" ring=compactor level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=cleaner run_id=1673979976 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=29.530981ms duration_ms=29 cached=3 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job="stage: split, range start: 0, range end: 7200000, shard: 1_of_1, blocks: 01GQ0FP56X8YZW2KYMTCT8C66H (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01GQ0FPBABATPY19SSRRN4BK9B (min time: 1970-01-01 00:01:00 +0000 UTC, max time: 1970-01-01 00:07:00 +0000 UTC),01GQ0FPH8Z46PNCTDT8EXHSKE6 (min time: 1970-01-01 01:00:00 +0000 UTC, max time: 1970-01-01 02:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=3 plan="[01GQ0FP56X8YZW2KYMTCT8C66H (min time: 0, max time: 300000) 01GQ0FPBABATPY19SSRRN4BK9B (min time: 60000, max time: 420000) 01GQ0FPH8Z46PNCTDT8EXHSKE6 (min time: 3600000, max time: 7200000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GQ0FPBABATPY19SSRRN4BK9B/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GQ0FP56X8YZW2KYMTCT8C66H/meta.json level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=51.539967ms level=info component=cleaner run_id=1673979976 msg="successfully completed blocks cleanup and maintenance" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GQ0FPH8Z46PNCTDT8EXHSKE6/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=3 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3462944219/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FP56X8YZW2KYMTCT8C66H /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3462944219/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FPBABATPY19SSRRN4BK9B /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3462944219/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FPH8Z46PNCTDT8EXHSKE6]" duration=20.327987ms duration_ms=20 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=3 mint=0 maxt=7200000 ulid=01GQ0FPPXFFPNT3DNYXZJA6N6P sources="[01GQ0FP56X8YZW2KYMTCT8C66H 01GQ0FPH8Z46PNCTDT8EXHSKE6 01GQ0FPBABATPY19SSRRN4BK9B]" duration=9.168741189s shard=1_of_2 level=info component=compactor msg="compact blocks" count=3 mint=0 maxt=7200000 ulid=01GQ0FPPXFAX139MRWFBHC8C7A sources="[01GQ0FP56X8YZW2KYMTCT8C66H 01GQ0FPH8Z46PNCTDT8EXHSKE6 01GQ0FPBABATPY19SSRRN4BK9B]" duration=9.169272188s shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compacted blocks" new="[01GQ0FPPXFFPNT3DNYXZJA6N6P 01GQ0FPPXFAX139MRWFBHC8C7A]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3462944219/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FP56X8YZW2KYMTCT8C66H /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3462944219/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FPBABATPY19SSRRN4BK9B /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3462944219/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FPH8Z46PNCTDT8EXHSKE6]" duration=9.169611188s duration_ms=9169 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3462944219/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FPPXFAX139MRWFBHC8C7A/chunks/000001 dst=01GQ0FPPXFAX139MRWFBHC8C7A/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3462944219/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3462944219/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FPPXFAX139MRWFBHC8C7A/index dst=01GQ0FPPXFAX139MRWFBHC8C7A/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3462944219/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded block" result_block=01GQ0FPPXFAX139MRWFBHC8C7A duration=3.384998ms duration_ms=3 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3462944219/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FPPXFFPNT3DNYXZJA6N6P/chunks/000001 dst=01GQ0FPPXFFPNT3DNYXZJA6N6P/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3462944219/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3462944219/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FPPXFFPNT3DNYXZJA6N6P/index dst=01GQ0FPPXFFPNT3DNYXZJA6N6P/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3462944219/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded block" result_block=01GQ0FPPXFFPNT3DNYXZJA6N6P duration=3.314998ms duration_ms=3 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=12.050992ms duration_ms=12 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GQ0FP56X8YZW2KYMTCT8C66H level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GQ0FP56X8YZW2KYMTCT8C66H level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GQ0FPBABATPY19SSRRN4BK9B level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GQ0FPBABATPY19SSRRN4BK9B level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GQ0FPH8Z46PNCTDT8EXHSKE6 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GQ0FPH8Z46PNCTDT8EXHSKE6 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compaction job succeeded" duration=9.209963162s duration_ms=9209 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=7.418995ms duration_ms=7 cached=5 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=10.225993ms duration_ms=10 cached=5 returned=2 partial=0 level=info component=compactor msg="lifecycler loop() exited gracefully" ring=compactor level=info component=compactor msg="changing instance state from" old_state=ACTIVE new_state=LEAVING ring=compactor level=info component=compactor msg="lifecycler entering final sleep before shutdown" final_sleep=0s level=debug component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the KV store" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="not loading tokens from file, tokens file path is empty" level=info component=compactor msg="instance not found in ring, adding with no tokens" ring=compactor level=debug component=compactor msg="JoinAfter expired" ring=compactor level=info component=compactor msg="auto-joining cluster after timeout" ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1673979997 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=8.433995ms duration_ms=8 cached=3 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job="stage: split, range start: 0, range end: 7200000, shard: 1_of_1, blocks: 01GQ0FQ00P7EM47069CX95CS5W (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01GQ0FQ3K6T9RFWRHVC9Q8G4MY (min time: 1970-01-01 00:01:00 +0000 UTC, max time: 1970-01-01 00:07:00 +0000 UTC),01GQ0FQ6RXM1V1AC5VFTX50W2X (min time: 1970-01-01 01:00:00 +0000 UTC, max time: 1970-01-01 02:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=3 plan="[01GQ0FQ00P7EM47069CX95CS5W (min time: 0, max time: 300000) 01GQ0FQ3K6T9RFWRHVC9Q8G4MY (min time: 60000, max time: 420000) 01GQ0FQ6RXM1V1AC5VFTX50W2X (min time: 3600000, max time: 7200000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GQ0FQ3K6T9RFWRHVC9Q8G4MY/meta.json level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=15.54799ms level=info component=cleaner run_id=1673979997 msg="successfully completed blocks cleanup and maintenance" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GQ0FQ6RXM1V1AC5VFTX50W2X/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GQ0FQ00P7EM47069CX95CS5W/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=3 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together1317077605/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FQ00P7EM47069CX95CS5W /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together1317077605/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FQ3K6T9RFWRHVC9Q8G4MY /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together1317077605/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FQ6RXM1V1AC5VFTX50W2X]" duration=32.229979ms duration_ms=32 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=3 mint=0 maxt=7200000 ulid=01GQ0FQBRBTH8TPPSQ4ZHGGRC0 sources="[01GQ0FQ6RXM1V1AC5VFTX50W2X 01GQ0FQ00P7EM47069CX95CS5W 01GQ0FQ3K6T9RFWRHVC9Q8G4MY]" duration=10.638954256s shard=1_of_2 level=info component=compactor msg="compact blocks" count=3 mint=0 maxt=7200000 ulid=01GQ0FQBRBRR0Z70ZJJEZM7K0V sources="[01GQ0FQ6RXM1V1AC5VFTX50W2X 01GQ0FQ00P7EM47069CX95CS5W 01GQ0FQ3K6T9RFWRHVC9Q8G4MY]" duration=10.642342254s shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compacted blocks" new="[01GQ0FQBRBTH8TPPSQ4ZHGGRC0 01GQ0FQBRBRR0Z70ZJJEZM7K0V]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together1317077605/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FQ00P7EM47069CX95CS5W /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together1317077605/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FQ3K6T9RFWRHVC9Q8G4MY /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together1317077605/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FQ6RXM1V1AC5VFTX50W2X]" duration=10.642914254s duration_ms=10642 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together1317077605/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FQBRBRR0Z70ZJJEZM7K0V/chunks/000001 dst=01GQ0FQBRBRR0Z70ZJJEZM7K0V/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together1317077605/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together1317077605/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FQBRBRR0Z70ZJJEZM7K0V/index dst=01GQ0FQBRBRR0Z70ZJJEZM7K0V/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together1317077605/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded block" result_block=01GQ0FQBRBRR0Z70ZJJEZM7K0V duration=4.221998ms duration_ms=4 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together1317077605/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FQBRBTH8TPPSQ4ZHGGRC0/chunks/000001 dst=01GQ0FQBRBTH8TPPSQ4ZHGGRC0/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together1317077605/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together1317077605/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FQBRBTH8TPPSQ4ZHGGRC0/index dst=01GQ0FQBRBTH8TPPSQ4ZHGGRC0/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together1317077605/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded block" result_block=01GQ0FQBRBTH8TPPSQ4ZHGGRC0 duration=10.497994ms duration_ms=10 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=18.340988ms duration_ms=18 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GQ0FQ00P7EM47069CX95CS5W level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GQ0FQ00P7EM47069CX95CS5W level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GQ0FQ3K6T9RFWRHVC9Q8G4MY level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GQ0FQ3K6T9RFWRHVC9Q8G4MY level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GQ0FQ6RXM1V1AC5VFTX50W2X level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GQ0FQ6RXM1V1AC5VFTX50W2X level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compaction job succeeded" duration=10.718923206s duration_ms=10718 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=19.404988ms duration_ms=19 cached=5 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=20.979986ms duration_ms=20 cached=5 returned=2 partial=0 level=info component=compactor msg="lifecycler loop() exited gracefully" ring=compactor level=info component=compactor msg="changing instance state from" old_state=ACTIVE new_state=LEAVING ring=compactor level=info component=compactor msg="lifecycler entering final sleep before shutdown" final_sleep=0s level=debug component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the KV store" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="not loading tokens from file, tokens file path is empty" level=info component=compactor msg="instance not found in ring, adding with no tokens" ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=debug component=compactor msg="JoinAfter expired" ring=compactor level=info component=compactor msg="auto-joining cluster after timeout" ring=compactor level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=info component=cleaner run_id=1673980018 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=10.729993ms level=info component=cleaner run_id=1673980018 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=13.369991ms duration_ms=13 cached=2 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job="stage: split, range start: 7200000, range end: 14400000, shard: 1_of_1, blocks: 01GQ0FQPE27M90BHDVMKYY1XTW (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC),01GQ0FQTZGSSM6CJ2VWAAJSXM8 (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GQ0FQPE27M90BHDVMKYY1XTW (min time: 7200000, max time: 14400000) 01GQ0FQTZGSSM6CJ2VWAAJSXM8 (min time: 7200000, max time: 14400000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GQ0FQPE27M90BHDVMKYY1XTW/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GQ0FQTZGSSM6CJ2VWAAJSXM8/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split1278992237/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GQ0FQPE27M90BHDVMKYY1XTW /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split1278992237/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GQ0FQTZGSSM6CJ2VWAAJSXM8]" duration=12.411992ms duration_ms=12 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01GQ0FR08NSG53GC2N7PV58WWS sources="[01GQ0FQPE27M90BHDVMKYY1XTW 01GQ0FQTZGSSM6CJ2VWAAJSXM8]" duration=8.205756799s shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01GQ0FR08N3ZJYQTG6TPM6R8JZ sources="[01GQ0FQPE27M90BHDVMKYY1XTW 01GQ0FQTZGSSM6CJ2VWAAJSXM8]" duration=8.206670798s shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new="[01GQ0FR08NSG53GC2N7PV58WWS 01GQ0FR08N3ZJYQTG6TPM6R8JZ]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split1278992237/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GQ0FQPE27M90BHDVMKYY1XTW /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split1278992237/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GQ0FQTZGSSM6CJ2VWAAJSXM8]" duration=8.210193796s duration_ms=8210 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split1278992237/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GQ0FR08N3ZJYQTG6TPM6R8JZ/chunks/000001 dst=01GQ0FR08N3ZJYQTG6TPM6R8JZ/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split1278992237/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split1278992237/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GQ0FR08N3ZJYQTG6TPM6R8JZ/index dst=01GQ0FR08N3ZJYQTG6TPM6R8JZ/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split1278992237/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GQ0FR08N3ZJYQTG6TPM6R8JZ duration=29.470982ms duration_ms=29 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split1278992237/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GQ0FR08NSG53GC2N7PV58WWS/chunks/000001 dst=01GQ0FR08NSG53GC2N7PV58WWS/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split1278992237/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split1278992237/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GQ0FR08NSG53GC2N7PV58WWS/index dst=01GQ0FR08NSG53GC2N7PV58WWS/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split1278992237/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GQ0FR08NSG53GC2N7PV58WWS duration=28.569982ms duration_ms=28 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=43.195972ms duration_ms=43 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GQ0FQPE27M90BHDVMKYY1XTW level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GQ0FQPE27M90BHDVMKYY1XTW level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GQ0FQTZGSSM6CJ2VWAAJSXM8 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GQ0FQTZGSSM6CJ2VWAAJSXM8 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=8.29858474s duration_ms=8298 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=9.754994ms duration_ms=9 cached=4 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=9.764994ms duration_ms=9 cached=4 returned=2 partial=0 level=info component=compactor msg="lifecycler loop() exited gracefully" ring=compactor level=info component=compactor msg="changing instance state from" old_state=ACTIVE new_state=LEAVING ring=compactor level=info component=compactor msg="lifecycler entering final sleep before shutdown" final_sleep=0s level=debug component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the KV store" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split level=info component=compactor msg="not loading tokens from file, tokens file path is empty" level=info component=compactor msg="instance not found in ring, adding with no tokens" ring=compactor level=debug component=compactor msg="JoinAfter expired" ring=compactor level=info component=compactor msg="auto-joining cluster after timeout" ring=compactor level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=info component=cleaner run_id=1673980043 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=15.216991ms level=info component=cleaner run_id=1673980043 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=18.880988ms duration_ms=18 cached=3 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job="stage: split, range start: 0, range end: 7200000, shard: 1_of_1, blocks: 01GQ0FR8RPX6AP15AJ64AEBSP3 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01GQ0FRDQK9V2C4K6REFTDK0WJ (min time: 1970-01-01 00:01:00 +0000 UTC, max time: 1970-01-01 00:07:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GQ0FR8RPX6AP15AJ64AEBSP3 (min time: 0, max time: 300000) 01GQ0FRDQK9V2C4K6REFTDK0WJ (min time: 60000, max time: 420000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GQ0FRDQK9V2C4K6REFTDK0WJ/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GQ0FR8RPX6AP15AJ64AEBSP3/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split505961017/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FR8RPX6AP15AJ64AEBSP3 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split505961017/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FRDQK9V2C4K6REFTDK0WJ]" duration=27.220982ms duration_ms=27 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=420000 ulid=01GQ0FRR8PTEQM71R7DC0TFPKC sources="[01GQ0FR8RPX6AP15AJ64AEBSP3 01GQ0FRDQK9V2C4K6REFTDK0WJ]" duration=7.810531049s shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=420000 ulid=01GQ0FRR8PEK16D0NEGNRSZPSN sources="[01GQ0FR8RPX6AP15AJ64AEBSP3 01GQ0FRDQK9V2C4K6REFTDK0WJ]" duration=7.810756049s shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="compacted blocks" new="[01GQ0FRR8PTEQM71R7DC0TFPKC 01GQ0FRR8PEK16D0NEGNRSZPSN]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split505961017/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FR8RPX6AP15AJ64AEBSP3 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split505961017/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FRDQK9V2C4K6REFTDK0WJ]" duration=7.811051049s duration_ms=7811 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split505961017/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FRR8PEK16D0NEGNRSZPSN/chunks/000001 dst=01GQ0FRR8PEK16D0NEGNRSZPSN/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split505961017/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split505961017/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FRR8PTEQM71R7DC0TFPKC/chunks/000001 dst=01GQ0FRR8PTEQM71R7DC0TFPKC/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split505961017/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split505961017/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FRR8PEK16D0NEGNRSZPSN/index dst=01GQ0FRR8PEK16D0NEGNRSZPSN/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split505961017/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split505961017/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FRR8PTEQM71R7DC0TFPKC/index dst=01GQ0FRR8PTEQM71R7DC0TFPKC/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split505961017/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded block" result_block=01GQ0FRR8PEK16D0NEGNRSZPSN duration=3.832997ms duration_ms=3 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded block" result_block=01GQ0FRR8PTEQM71R7DC0TFPKC duration=5.760996ms duration_ms=5 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=8.784994ms duration_ms=8 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GQ0FR8RPX6AP15AJ64AEBSP3 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="block has been marked for deletion" block=01GQ0FR8RPX6AP15AJ64AEBSP3 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GQ0FRDQK9V2C4K6REFTDK0WJ level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="block has been marked for deletion" block=01GQ0FRDQK9V2C4K6REFTDK0WJ level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="compaction job succeeded" duration=7.85628202s duration_ms=7856 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=5.953996ms duration_ms=5 cached=5 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=10.690993ms duration_ms=10 cached=5 returned=3 partial=0 level=info component=compactor msg="lifecycler loop() exited gracefully" ring=compactor level=info component=compactor msg="changing instance state from" old_state=ACTIVE new_state=LEAVING ring=compactor level=info component=compactor msg="lifecycler entering final sleep before shutdown" final_sleep=0s level=debug component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the KV store" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/non-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="not loading tokens from file, tokens file path is empty" level=info component=compactor msg="instance not found in ring, adding with no tokens" ring=compactor level=debug component=compactor msg="JoinAfter expired" ring=compactor level=info component=compactor msg="auto-joining cluster after timeout" ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=info component=cleaner run_id=1673980062 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=6.578995ms level=info component=cleaner run_id=1673980062 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=11.711993ms duration_ms=11 cached=3 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job="stage: split, range start: 0, range end: 7200000, shard: 1_of_1, blocks: 01GQ0FS017MEYG424QF9HEZ189 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01GQ0FS36DC1ADSNC0G76QQKY1 (min time: 1970-01-01 00:05:00 +0000 UTC, max time: 1970-01-01 00:10:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GQ0FS017MEYG424QF9HEZ189 (min time: 0, max time: 300000) 01GQ0FS36DC1ADSNC0G76QQKY1 (min time: 300000, max time: 600000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GQ0FS36DC1ADSNC0G76QQKY1/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GQ0FS017MEYG424QF9HEZ189/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split3374136764/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FS017MEYG424QF9HEZ189 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split3374136764/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FS36DC1ADSNC0G76QQKY1]" duration=11.283993ms duration_ms=11 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=600000 ulid=01GQ0FSAMDBHTXFG20RB1XVJ6Z sources="[01GQ0FS36DC1ADSNC0G76QQKY1 01GQ0FS017MEYG424QF9HEZ189]" duration=5.425473561s shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=600000 ulid=01GQ0FSAMDWZSJN5Y0NR5RDJAV sources="[01GQ0FS36DC1ADSNC0G76QQKY1 01GQ0FS017MEYG424QF9HEZ189]" duration=5.425672561s shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compacted blocks" new="[01GQ0FSAMDBHTXFG20RB1XVJ6Z 01GQ0FSAMDWZSJN5Y0NR5RDJAV]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split3374136764/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FS017MEYG424QF9HEZ189 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split3374136764/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FS36DC1ADSNC0G76QQKY1]" duration=5.425941561s duration_ms=5425 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split3374136764/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FSAMDWZSJN5Y0NR5RDJAV/chunks/000001 dst=01GQ0FSAMDWZSJN5Y0NR5RDJAV/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split3374136764/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split3374136764/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FSAMDWZSJN5Y0NR5RDJAV/index dst=01GQ0FSAMDWZSJN5Y0NR5RDJAV/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split3374136764/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded block" result_block=01GQ0FSAMDWZSJN5Y0NR5RDJAV duration=3.474998ms duration_ms=3 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split3374136764/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FSAMDBHTXFG20RB1XVJ6Z/chunks/000001 dst=01GQ0FSAMDBHTXFG20RB1XVJ6Z/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split3374136764/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split3374136764/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GQ0FSAMDBHTXFG20RB1XVJ6Z/index dst=01GQ0FSAMDBHTXFG20RB1XVJ6Z/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split3374136764/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded block" result_block=01GQ0FSAMDBHTXFG20RB1XVJ6Z duration=5.470996ms duration_ms=5 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=7.996995ms duration_ms=7 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GQ0FS017MEYG424QF9HEZ189 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01GQ0FS017MEYG424QF9HEZ189 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GQ0FS36DC1ADSNC0G76QQKY1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01GQ0FS36DC1ADSNC0G76QQKY1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction job succeeded" duration=5.450201545s duration_ms=5450 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=6.929996ms duration_ms=6 cached=5 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=9.372994ms duration_ms=9 cached=5 returned=3 partial=0 level=info component=compactor msg="lifecycler loop() exited gracefully" ring=compactor level=info component=compactor msg="changing instance state from" old_state=ACTIVE new_state=LEAVING ring=compactor level=info component=compactor msg="lifecycler entering final sleep before shutdown" final_sleep=0s level=debug component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the KV store" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/splitting_should_be_disabled_if_configured_shards_=_0 level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="not loading tokens from file, tokens file path is empty" level=info component=compactor msg="instance not found in ring, adding with no tokens" ring=compactor level=debug component=compactor msg="JoinAfter expired" ring=compactor level=info component=compactor msg="auto-joining cluster after timeout" ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1673980079 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=7.607995ms duration_ms=7 cached=3 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge--0-7200000 job="stage: merge, range start: 0, range end: 7200000, shard: , blocks: 01GQ0FSG3W0NGH5HQHRB4BTZ4A (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01GQ0FSK7ZEVNRZ6NJ3QS0T1YY (min time: 1970-01-01 00:05:00 +0000 UTC, max time: 1970-01-01 00:10:00 +0000 UTC)" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=7.323995ms level=info component=cleaner run_id=1673980079 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GQ0FSG3W0NGH5HQHRB4BTZ4A (min time: 0, max time: 300000) 01GQ0FSK7ZEVNRZ6NJ3QS0T1YY (min time: 300000, max time: 600000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GQ0FSK7ZEVNRZ6NJ3QS0T1YY/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GQ0FSG3W0NGH5HQHRB4BTZ4A/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_01522310641/001/compact/0@17241709254077376921-merge--0-7200000/01GQ0FSG3W0NGH5HQHRB4BTZ4A /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_01522310641/001/compact/0@17241709254077376921-merge--0-7200000/01GQ0FSK7ZEVNRZ6NJ3QS0T1YY]" duration=10.085994ms duration_ms=10 level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=600000 ulid=01GQ0FSVWSPSNKWDXWSBXS0174 sources="[01GQ0FSG3W0NGH5HQHRB4BTZ4A 01GQ0FSK7ZEVNRZ6NJ3QS0T1YY]" duration=4.03798644s shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compacted blocks" new=[01GQ0FSVWSPSNKWDXWSBXS0174] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_01522310641/001/compact/0@17241709254077376921-merge--0-7200000/01GQ0FSG3W0NGH5HQHRB4BTZ4A /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_01522310641/001/compact/0@17241709254077376921-merge--0-7200000/01GQ0FSK7ZEVNRZ6NJ3QS0T1YY]" duration=4.03857444s duration_ms=4038 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_01522310641/001/compact/0@17241709254077376921-merge--0-7200000/01GQ0FSVWSPSNKWDXWSBXS0174/chunks/000001" dst=01GQ0FSVWSPSNKWDXWSBXS0174/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_01522310641/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_01522310641/001/compact/0@17241709254077376921-merge--0-7200000/01GQ0FSVWSPSNKWDXWSBXS0174/index" dst=01GQ0FSVWSPSNKWDXWSBXS0174/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_01522310641/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded block" result_block=01GQ0FSVWSPSNKWDXWSBXS0174 duration=4.772997ms duration_ms=4 external_labels={} level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=8.296995ms duration_ms=8 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GQ0FSG3W0NGH5HQHRB4BTZ4A level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01GQ0FSG3W0NGH5HQHRB4BTZ4A level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GQ0FSK7ZEVNRZ6NJ3QS0T1YY level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01GQ0FSK7ZEVNRZ6NJ3QS0T1YY level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction job succeeded" duration=4.065469423s duration_ms=4065 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=6.469996ms duration_ms=6 cached=4 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=7.290995ms duration_ms=7 cached=4 returned=2 partial=0 level=info component=compactor msg="lifecycler loop() exited gracefully" ring=compactor level=info component=compactor msg="changing instance state from" old_state=ACTIVE new_state=LEAVING ring=compactor level=info component=compactor msg="lifecycler entering final sleep before shutdown" final_sleep=0s level=debug component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the KV store" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/splitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0 level=info component=compactor msg="not loading tokens from file, tokens file path is empty" level=info component=compactor msg="instance not found in ring, adding with no tokens" ring=compactor level=debug component=compactor msg="JoinAfter expired" ring=compactor level=info component=compactor msg="auto-joining cluster after timeout" ring=compactor level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=cleaner run_id=1673980123 msg="started blocks cleanup and maintenance" level=info component=compactor user=user-1 msg="start sync of metas" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=19.393988ms duration_ms=19 cached=10 returned=10 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job="stage: merge, range start: 14400000, range end: 28800000, shard: 1_of_2, blocks: 01GQ0FTDGCEKBSXZDK8TPV2W8S (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 06:00:00 +0000 UTC),01GQ0FTMXBMGXBH8F4C67S79MM (min time: 1970-01-01 06:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job="stage: merge, range start: 14400000, range end: 28800000, shard: 2_of_2, blocks: 01GQ0FTGQ3SXB903T9Y54B0930 (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 06:00:00 +0000 UTC),01GQ0FTSAQRZPSHHTZJ1Z3D19W (min time: 1970-01-01 06:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge--28800000-57600000 job="stage: merge, range start: 28800000, range end: 57600000, shard: , blocks: 01GQ0FTYR5P81PT5S4EJ4AN0M0 (min time: 1970-01-01 08:00:00 +0000 UTC, max time: 1970-01-01 10:00:00 +0000 UTC),01GQ0FV2PNX9FCX96A5SBPGSGK (min time: 1970-01-01 14:00:00 +0000 UTC, max time: 1970-01-01 16:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge--7200000-14400000 job="stage: merge, range start: 7200000, range end: 14400000, shard: , blocks: 01GQ0FT72NXTF9XFYXJ14NX6XM (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC),01GQ0FTA4RAGFVSQA0GXDDBHJA (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GQ0FT72NXTF9XFYXJ14NX6XM (min time: 7200000, max time: 14400000) 01GQ0FTA4RAGFVSQA0GXDDBHJA (min time: 7200000, max time: 14400000)]" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=24.466985ms level=info component=cleaner run_id=1673980123 msg="successfully completed blocks cleanup and maintenance" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GQ0FTA4RAGFVSQA0GXDDBHJA/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GQ0FT72NXTF9XFYXJ14NX6XM/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0962745273/001/compact/0@17241709254077376921-merge--7200000-14400000/01GQ0FT72NXTF9XFYXJ14NX6XM /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0962745273/001/compact/0@17241709254077376921-merge--7200000-14400000/01GQ0FTA4RAGFVSQA0GXDDBHJA]" duration=11.796993ms duration_ms=11 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01GQ0FV6AC6N39TT6X0AAJQ75R sources="[01GQ0FTA4RAGFVSQA0GXDDBHJA 01GQ0FT72NXTF9XFYXJ14NX6XM]" duration=3.718662643s shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new=[01GQ0FV6AC6N39TT6X0AAJQ75R] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0962745273/001/compact/0@17241709254077376921-merge--7200000-14400000/01GQ0FT72NXTF9XFYXJ14NX6XM /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0962745273/001/compact/0@17241709254077376921-merge--7200000-14400000/01GQ0FTA4RAGFVSQA0GXDDBHJA]" duration=3.719013642s duration_ms=3719 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0962745273/001/compact/0@17241709254077376921-merge--7200000-14400000/01GQ0FV6AC6N39TT6X0AAJQ75R/chunks/000001" dst=01GQ0FV6AC6N39TT6X0AAJQ75R/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0962745273/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0962745273/001/compact/0@17241709254077376921-merge--7200000-14400000/01GQ0FV6AC6N39TT6X0AAJQ75R/index" dst=01GQ0FV6AC6N39TT6X0AAJQ75R/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0962745273/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GQ0FV6AC6N39TT6X0AAJQ75R duration=4.091997ms duration_ms=4 external_labels={} level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=17.147989ms duration_ms=17 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GQ0FT72NXTF9XFYXJ14NX6XM level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GQ0FT72NXTF9XFYXJ14NX6XM level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GQ0FTA4RAGFVSQA0GXDDBHJA level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GQ0FTA4RAGFVSQA0GXDDBHJA level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=3.753197621s duration_ms=3753 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GQ0FTDGCEKBSXZDK8TPV2W8S (min time: 14400000, max time: 21600000) 01GQ0FTMXBMGXBH8F4C67S79MM (min time: 21600000, max time: 28800000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GQ0FTMXBMGXBH8F4C67S79MM/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GQ0FTDGCEKBSXZDK8TPV2W8S/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0962745273/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GQ0FTDGCEKBSXZDK8TPV2W8S /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0962745273/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GQ0FTMXBMGXBH8F4C67S79MM]" duration=12.178992ms duration_ms=12 level=info component=compactor msg="compact blocks" count=2 mint=14400000 maxt=28800000 ulid=01GQ0FVA017FQS140BS8T07BEN sources="[01GQ0FTDGCEKBSXZDK8TPV2W8S 01GQ0FTMXBMGXBH8F4C67S79MM]" duration=3.403463842s shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compacted blocks" new=[01GQ0FVA017FQS140BS8T07BEN] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0962745273/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GQ0FTDGCEKBSXZDK8TPV2W8S /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0962745273/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GQ0FTMXBMGXBH8F4C67S79MM]" duration=3.403912842s duration_ms=3403 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0962745273/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GQ0FVA017FQS140BS8T07BEN/chunks/000001" dst=01GQ0FVA017FQS140BS8T07BEN/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0962745273/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0962745273/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GQ0FVA017FQS140BS8T07BEN/index" dst=01GQ0FVA017FQS140BS8T07BEN/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0962745273/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded block" result_block=01GQ0FVA017FQS140BS8T07BEN duration=3.721998ms duration_ms=3 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=6.589996ms duration_ms=6 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GQ0FTDGCEKBSXZDK8TPV2W8S level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GQ0FTDGCEKBSXZDK8TPV2W8S level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GQ0FTMXBMGXBH8F4C67S79MM level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GQ0FTMXBMGXBH8F4C67S79MM level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction job succeeded" duration=3.441237819s duration_ms=3441 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GQ0FTGQ3SXB903T9Y54B0930 (min time: 14400000, max time: 21600000) 01GQ0FTSAQRZPSHHTZJ1Z3D19W (min time: 21600000, max time: 28800000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GQ0FTGQ3SXB903T9Y54B0930/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GQ0FTSAQRZPSHHTZJ1Z3D19W/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0962745273/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GQ0FTGQ3SXB903T9Y54B0930 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0962745273/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GQ0FTSAQRZPSHHTZJ1Z3D19W]" duration=10.940993ms duration_ms=10 level=info component=compactor msg="compact blocks" count=2 mint=14400000 maxt=28800000 ulid=01GQ0FVDB8G4ECMWGAW3ZMYBGQ sources="[01GQ0FTGQ3SXB903T9Y54B0930 01GQ0FTSAQRZPSHHTZJ1Z3D19W]" duration=5.450020545s shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compacted blocks" new=[01GQ0FVDB8G4ECMWGAW3ZMYBGQ] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0962745273/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GQ0FTGQ3SXB903T9Y54B0930 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0962745273/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GQ0FTSAQRZPSHHTZJ1Z3D19W]" duration=5.450608545s duration_ms=5450 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0962745273/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GQ0FVDB8G4ECMWGAW3ZMYBGQ/chunks/000001" dst=01GQ0FVDB8G4ECMWGAW3ZMYBGQ/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0962745273/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0962745273/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GQ0FVDB8G4ECMWGAW3ZMYBGQ/index" dst=01GQ0FVDB8G4ECMWGAW3ZMYBGQ/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0962745273/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded block" result_block=01GQ0FVDB8G4ECMWGAW3ZMYBGQ duration=3.832997ms duration_ms=3 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=6.999995ms duration_ms=6 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GQ0FTGQ3SXB903T9Y54B0930 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GQ0FTGQ3SXB903T9Y54B0930 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GQ0FTSAQRZPSHHTZJ1Z3D19W level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GQ0FTSAQRZPSHHTZJ1Z3D19W level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction job succeeded" duration=5.47354753s duration_ms=5473 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GQ0FTYR5P81PT5S4EJ4AN0M0 (min time: 28800000, max time: 36000000) 01GQ0FV2PNX9FCX96A5SBPGSGK (min time: 50400000, max time: 57600000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GQ0FTYR5P81PT5S4EJ4AN0M0/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GQ0FV2PNX9FCX96A5SBPGSGK/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0962745273/001/compact/0@17241709254077376921-merge--28800000-57600000/01GQ0FTYR5P81PT5S4EJ4AN0M0 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0962745273/001/compact/0@17241709254077376921-merge--28800000-57600000/01GQ0FV2PNX9FCX96A5SBPGSGK]" duration=20.947987ms duration_ms=20 split_merge_compactor_test.go:640: expected , got Diff: --- metric output does not match expectation; want +++ got: @@ -2,3 +2,3 @@ # TYPE cortex_compactor_runs_completed_total counter -cortex_compactor_runs_completed_total 1 +cortex_compactor_runs_completed_total 0 level=info component=compactor msg="lifecycler loop() exited gracefully" ring=compactor level=info component=compactor msg="changing instance state from" old_state=ACTIVE new_state=LEAVING ring=compactor level=info component=compactor msg="lifecycler entering final sleep before shutdown" final_sleep=0s level=debug component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the KV store" ring=compactor level=error component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compaction job failed" duration=4.209333331s duration_ms=4209 err="compact blocks [/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0962745273/001/compact/0@17241709254077376921-merge--28800000-57600000/01GQ0FTYR5P81PT5S4EJ4AN0M0 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0962745273/001/compact/0@17241709254077376921-merge--28800000-57600000/01GQ0FV2PNX9FCX96A5SBPGSGK]: 2 errors: populate block: adding series: asyncBlockWriter doesn't run anymore: add series: context canceled; context canceled" level=error component=compactor msg="failed to compact user blocks" user=user-1 err="compaction: group 0@17241709254077376921-merge--28800000-57600000: compact blocks [/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0962745273/001/compact/0@17241709254077376921-merge--28800000-57600000/01GQ0FTYR5P81PT5S4EJ4AN0M0 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0962745273/001/compact/0@17241709254077376921-merge--28800000-57600000/01GQ0FV2PNX9FCX96A5SBPGSGK]: 2 errors: populate block: adding series: asyncBlockWriter doesn't run anymore: add series: context canceled; context canceled" --- FAIL: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor (315.09s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/should_correctly_handle_empty_blocks_generated_in_the_splitting_stage (8.80s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split (13.92s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/non-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split (27.15s) --- FAIL: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/smaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges (83.21s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together (27.54s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together (22.89s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split (18.73s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split (23.90s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/non-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split (16.48s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/splitting_should_be_disabled_if_configured_shards_=_0 (16.36s) --- FAIL: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/splitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0 (56.09s) === RUN TestPlanCompaction === RUN TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_in_different_time_ranges,_two_split_groups split_merge_grouper_test.go:515: got 4 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 10, shard: 1_of_2, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 10, shard: 2_of_2, blocks: 00000000020000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_2, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 2_of_2, blocks: 00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/should_merge_but_NOT_split_multiple_1st_level_blocks_in_different_time_ranges_if_configured_shards_=_0 split_merge_grouper_test.go:515: got 2 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 0, range end: 10, shard: , blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC) split_merge_grouper_test.go:517: - stage: merge, range start: 10, range end: 20, shard: , blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/should_merge_split_blocks_that_can_be_compacted_on_the_2nd_range_only split_merge_grouper_test.go:515: got 3 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 0, range end: 20, shard: 1_of_2, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: merge, range start: 0, range end: 20, shard: 2_of_2, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: merge, range start: 20, range end: 40, shard: 1_of_2, blocks: 00000000050000000000000000 (min time: 1970-01-01 00:00:00.02 +0000 UTC, max time: 1970-01-01 00:00:00.03 +0000 UTC),00000000060000000000000000 (min time: 1970-01-01 00:00:00.03 +0000 UTC, max time: 1970-01-01 00:00:00.04 +0000 UTC) === RUN TestPlanCompaction/a_block_with_time_range_crossing_two_1st_level_ranges_should_BE_considered_for_2nd_level_compaction split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 0, range end: 40, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.03 +0000 UTC),00000000030000000000000000 (min time: 1970-01-01 00:00:00.02 +0000 UTC, max time: 1970-01-01 00:00:00.04 +0000 UTC) === RUN TestPlanCompaction/should_split_a_single_block_if_==_smallest_compaction_range split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 20, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/should_NOT_split_a_single_block_if_==_smallest_compaction_range_but_configured_shards_=_0 split_merge_grouper_test.go:515: got 0 jobs: === RUN TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_in_different_time_ranges,_single_split_group split_merge_grouper_test.go:515: got 2 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 10, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_1, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/should_not_split_non-split_blocks_if_they're_>_smallest_compaction_range_(do_not_split_historical_blocks_after_enabling_splitting) split_merge_grouper_test.go:515: got 2 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 0, range end: 20, shard: 1_of_2, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: merge, range start: 0, range end: 20, shard: 2_of_2, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/input_blocks_can_be_compacted_on_a_mix_of_1st_and_2nd_ranges,_guaranteeing_no_overlaps_and_giving_preference_to_smaller_ranges split_merge_grouper_test.go:515: got 3 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 20, range end: 40, shard: 1_of_1, blocks: 00000000040000000000000000 (min time: 1970-01-01 00:00:00.02 +0000 UTC, max time: 1970-01-01 00:00:00.03 +0000 UTC),00000000050000000000000000 (min time: 1970-01-01 00:00:00.03 +0000 UTC, max time: 1970-01-01 00:00:00.04 +0000 UTC) split_merge_grouper_test.go:517: - stage: merge, range start: 70, range end: 80, shard: 1_of_1, blocks: 00000000080000000000000000 (min time: 1970-01-01 00:00:00.07 +0000 UTC, max time: 1970-01-01 00:00:00.08 +0000 UTC),00000000090000000000000000 (min time: 1970-01-01 00:00:00.075 +0000 UTC, max time: 1970-01-01 00:00:00.08 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 10, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00.007 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC) === RUN TestPlanCompaction/input_blocks_have_already_been_compacted_with_the_largest_range split_merge_grouper_test.go:515: got 0 jobs: === RUN TestPlanCompaction/a_block_with_time_range_larger_then_the_largest_compaction_range_should_NOT_be_considered_for_compaction split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 80, range end: 120, shard: 1_of_1, blocks: 00000000040000000000000000 (min time: 1970-01-01 00:00:00.08 +0000 UTC, max time: 1970-01-01 00:00:00.12 +0000 UTC),00000000050000000000000000 (min time: 1970-01-01 00:00:00.08 +0000 UTC, max time: 1970-01-01 00:00:00.12 +0000 UTC) === RUN TestPlanCompaction/a_block_with_time_range_crossing_two_1st_level_ranges_should_be_NOT_considered_for_1st_level_splitting split_merge_grouper_test.go:515: got 2 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 20, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 20, range end: 40, shard: 1_of_1, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00.02 +0000 UTC, max time: 1970-01-01 00:00:00.03 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.03 +0000 UTC, max time: 1970-01-01 00:00:00.04 +0000 UTC) === RUN TestPlanCompaction/a_range_containing_the_most_recent_block_shouldn't_be_prematurely_compacted_if_doesn't_cover_the_full_range split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 10, shard: 1_of_1, blocks: 00000000000000000000000000 (min time: 1970-01-01 00:00:00.005 +0000 UTC, max time: 1970-01-01 00:00:00.008 +0000 UTC),00000000000000000000000000 (min time: 1970-01-01 00:00:00.007 +0000 UTC, max time: 1970-01-01 00:00:00.009 +0000 UTC) === RUN TestPlanCompaction/should_not_merge_blocks_within_the_same_time_range_but_with_different_external_labels split_merge_grouper_test.go:515: got 3 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_1, blocks: 00000000020000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC),00000000030000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_1, blocks: 00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/no_input_blocks split_merge_grouper_test.go:515: got 0 jobs: === RUN TestPlanCompaction/should_split_a_single_block_if_<_smallest_compaction_range split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 20, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_within_the_same_time_range split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_in_different_time_ranges split_merge_grouper_test.go:515: got 2 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 10, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_1, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/input_blocks_match_the_largest_range_but_can_be_compacted_because_overlapping split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 80, range end: 120, shard: 1_of_1, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00.08 +0000 UTC, max time: 1970-01-01 00:00:00.12 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.08 +0000 UTC, max time: 1970-01-01 00:00:00.12 +0000 UTC) --- PASS: TestPlanCompaction (0.02s) --- PASS: TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_in_different_time_ranges,_two_split_groups (0.00s) --- PASS: TestPlanCompaction/should_merge_but_NOT_split_multiple_1st_level_blocks_in_different_time_ranges_if_configured_shards_=_0 (0.00s) --- PASS: TestPlanCompaction/should_merge_split_blocks_that_can_be_compacted_on_the_2nd_range_only (0.00s) --- PASS: TestPlanCompaction/a_block_with_time_range_crossing_two_1st_level_ranges_should_BE_considered_for_2nd_level_compaction (0.00s) --- PASS: TestPlanCompaction/should_split_a_single_block_if_==_smallest_compaction_range (0.00s) --- PASS: TestPlanCompaction/should_NOT_split_a_single_block_if_==_smallest_compaction_range_but_configured_shards_=_0 (0.00s) --- PASS: TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_in_different_time_ranges,_single_split_group (0.00s) --- PASS: TestPlanCompaction/should_not_split_non-split_blocks_if_they're_>_smallest_compaction_range_(do_not_split_historical_blocks_after_enabling_splitting) (0.00s) --- PASS: TestPlanCompaction/input_blocks_can_be_compacted_on_a_mix_of_1st_and_2nd_ranges,_guaranteeing_no_overlaps_and_giving_preference_to_smaller_ranges (0.00s) --- PASS: TestPlanCompaction/input_blocks_have_already_been_compacted_with_the_largest_range (0.00s) --- PASS: TestPlanCompaction/a_block_with_time_range_larger_then_the_largest_compaction_range_should_NOT_be_considered_for_compaction (0.00s) --- PASS: TestPlanCompaction/a_block_with_time_range_crossing_two_1st_level_ranges_should_be_NOT_considered_for_1st_level_splitting (0.00s) --- PASS: TestPlanCompaction/a_range_containing_the_most_recent_block_shouldn't_be_prematurely_compacted_if_doesn't_cover_the_full_range (0.00s) --- PASS: TestPlanCompaction/should_not_merge_blocks_within_the_same_time_range_but_with_different_external_labels (0.00s) --- PASS: TestPlanCompaction/no_input_blocks (0.00s) --- PASS: TestPlanCompaction/should_split_a_single_block_if_<_smallest_compaction_range (0.00s) --- PASS: TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_within_the_same_time_range (0.00s) --- PASS: TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_in_different_time_ranges (0.00s) --- PASS: TestPlanCompaction/input_blocks_match_the_largest_range_but_can_be_compacted_because_overlapping (0.00s) === RUN TestPlanSplitting === RUN TestPlanSplitting/should_return_nil_if_the_input_group_contains_no_non-sharded_blocks === RUN TestPlanSplitting/should_return_a_split_job_if_the_input_group_contains_1_non-sharded_block === RUN TestPlanSplitting/should_splitGroups_split_jobs_if_the_input_group_contains_multiple_non-sharded_blocks === RUN TestPlanSplitting/should_return_nil_if_the_input_group_is_empty --- PASS: TestPlanSplitting (0.00s) --- PASS: TestPlanSplitting/should_return_nil_if_the_input_group_contains_no_non-sharded_blocks (0.00s) --- PASS: TestPlanSplitting/should_return_a_split_job_if_the_input_group_contains_1_non-sharded_block (0.00s) --- PASS: TestPlanSplitting/should_splitGroups_split_jobs_if_the_input_group_contains_multiple_non-sharded_blocks (0.00s) --- PASS: TestPlanSplitting/should_return_nil_if_the_input_group_is_empty (0.00s) === RUN TestGroupBlocksByShardID === RUN TestGroupBlocksByShardID/no_input_blocks === RUN TestGroupBlocksByShardID/only_1_block_in_input_with_shard_ID === RUN TestGroupBlocksByShardID/only_1_block_in_input_without_shard_ID === RUN TestGroupBlocksByShardID/multiple_blocks_per_shard_ID --- PASS: TestGroupBlocksByShardID (0.00s) --- PASS: TestGroupBlocksByShardID/no_input_blocks (0.00s) --- PASS: TestGroupBlocksByShardID/only_1_block_in_input_with_shard_ID (0.00s) --- PASS: TestGroupBlocksByShardID/only_1_block_in_input_without_shard_ID (0.00s) --- PASS: TestGroupBlocksByShardID/multiple_blocks_per_shard_ID (0.00s) === RUN TestGroupBlocksByRange === RUN TestGroupBlocksByRange/multiple_blocks_per_range === RUN TestGroupBlocksByRange/a_block_with_time_range_larger_then_the_range_should_be_excluded === RUN TestGroupBlocksByRange/blocks_with_different_time_ranges_but_all_fitting_within_the_input_range === RUN TestGroupBlocksByRange/no_input_blocks === RUN TestGroupBlocksByRange/only_1_block_in_input === RUN TestGroupBlocksByRange/only_1_block_per_range --- PASS: TestGroupBlocksByRange (0.00s) --- PASS: TestGroupBlocksByRange/multiple_blocks_per_range (0.00s) --- PASS: TestGroupBlocksByRange/a_block_with_time_range_larger_then_the_range_should_be_excluded (0.00s) --- PASS: TestGroupBlocksByRange/blocks_with_different_time_ranges_but_all_fitting_within_the_input_range (0.00s) --- PASS: TestGroupBlocksByRange/no_input_blocks (0.00s) --- PASS: TestGroupBlocksByRange/only_1_block_in_input (0.00s) --- PASS: TestGroupBlocksByRange/only_1_block_per_range (0.00s) === RUN TestJob_conflicts === RUN TestJob_conflicts/should_conflict_if_jobs_compact_same_blocks_with_overlapping_time_ranges_and_different_shard_but_at_a_different_stage === RUN TestJob_conflicts/should_conflict_between_split_and_merge_jobs_with_overlapping_time_ranges === RUN TestJob_conflicts/should_NOT_conflict_between_split_and_merge_jobs_with_non-overlapping_time_ranges === RUN TestJob_conflicts/should_conflict_if_jobs_compact_different_blocks_but_with_overlapping_time_ranges_and_same_shard === RUN TestJob_conflicts/should_NOT_conflict_if_jobs_compact_different_blocks_with_non-overlapping_time_ranges_and_same_shard === RUN TestJob_conflicts/should_NOT_conflict_if_jobs_compact_same_blocks_with_overlapping_time_ranges_but_different_shard --- PASS: TestJob_conflicts (0.00s) --- PASS: TestJob_conflicts/should_conflict_if_jobs_compact_same_blocks_with_overlapping_time_ranges_and_different_shard_but_at_a_different_stage (0.00s) --- PASS: TestJob_conflicts/should_conflict_between_split_and_merge_jobs_with_overlapping_time_ranges (0.00s) --- PASS: TestJob_conflicts/should_NOT_conflict_between_split_and_merge_jobs_with_non-overlapping_time_ranges (0.00s) --- PASS: TestJob_conflicts/should_conflict_if_jobs_compact_different_blocks_but_with_overlapping_time_ranges_and_same_shard (0.00s) --- PASS: TestJob_conflicts/should_NOT_conflict_if_jobs_compact_different_blocks_with_non-overlapping_time_ranges_and_same_shard (0.00s) --- PASS: TestJob_conflicts/should_NOT_conflict_if_jobs_compact_same_blocks_with_overlapping_time_ranges_but_different_shard (0.00s) === RUN TestBlocksGroup_overlaps --- PASS: TestBlocksGroup_overlaps (0.00s) === RUN TestBlocksGroup_getNonShardedBlocks --- PASS: TestBlocksGroup_getNonShardedBlocks (0.00s) === RUN TestSplitAndMergePlanner_Plan === RUN TestSplitAndMergePlanner_Plan/source_blocks_are_smaller_then_the_largest_range_but_compacted_block_is_larger === RUN TestSplitAndMergePlanner_Plan/source_blocks_and_compacted_block_are_smaller_then_the_largest_range_but_misaligned === RUN TestSplitAndMergePlanner_Plan/blocks_fit_within_the_largest_range === RUN TestSplitAndMergePlanner_Plan/no_blocks === RUN TestSplitAndMergePlanner_Plan/a_source_block_is_larger_then_the_largest_range --- PASS: TestSplitAndMergePlanner_Plan (0.00s) --- PASS: TestSplitAndMergePlanner_Plan/source_blocks_are_smaller_then_the_largest_range_but_compacted_block_is_larger (0.00s) --- PASS: TestSplitAndMergePlanner_Plan/source_blocks_and_compacted_block_are_smaller_then_the_largest_range_but_misaligned (0.00s) --- PASS: TestSplitAndMergePlanner_Plan/blocks_fit_within_the_largest_range (0.00s) --- PASS: TestSplitAndMergePlanner_Plan/no_blocks (0.00s) --- PASS: TestSplitAndMergePlanner_Plan/a_source_block_is_larger_then_the_largest_range (0.00s) === RUN TestSyncerMetrics --- PASS: TestSyncerMetrics (0.01s) === RUN TestDeleteTenant --- PASS: TestDeleteTenant (0.14s) === RUN TestDeleteTenantStatus === RUN TestDeleteTenantStatus/empty === RUN TestDeleteTenantStatus/no_user_objects === RUN TestDeleteTenantStatus/non-block_files === RUN TestDeleteTenantStatus/block_files --- PASS: TestDeleteTenantStatus (0.49s) --- PASS: TestDeleteTenantStatus/empty (0.16s) --- PASS: TestDeleteTenantStatus/no_user_objects (0.13s) --- PASS: TestDeleteTenantStatus/non-block_files (0.02s) --- PASS: TestDeleteTenantStatus/block_files (0.17s) === CONT TestSyncer_GarbageCollect_e2e === RUN TestSyncer_GarbageCollect_e2e/inmem create 00000000000000000000000000 create 00000000010000000000000000 create 00000000020000000000000000 create 00000000030000000000000000 create 00000000040000000000000000 create 00000000050000000000000000 create 00000000060000000000000000 create 00000000070000000000000000 create 00000000080000000000000000 create 00000000090000000000000000 create 00000000340000000000000000 create 00000000680000000000000000 create 000000009C0000000000000000 create 00000000CG0000000000000000 create 00000000FM0000000000000000 === CONT TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForDeletion === CONT TestMultitenantCompactor_ShouldRetryCompactionOnFailureWhileDiscoveringUsersFromBucket === CONT TestMultitenantCompactor_ShouldCompactOnlyUsersOwnedByTheInstanceOnShardingEnabledAndMultipleInstancesRunning === RUN TestSyncer_GarbageCollect_e2e/filesystem === PAUSE TestSyncer_GarbageCollect_e2e/filesystem === CONT TestMultitenantCompactor_ShouldFailCompactionOnTimeout --- PASS: TestMultitenantCompactor_ShouldRetryCompactionOnFailureWhileDiscoveringUsersFromBucket (0.17s) === CONT TestMultitenantCompactor_ShouldSkipCompactionForJobsNoMoreOwnedAfterPlanning --- PASS: TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForDeletion (0.31s) === CONT TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly === RUN TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly/inmem create 00000000000000000000000000 create 00000000010000000000000000 create 00000000340000000000000000 level=debug msg="deleted file" file=00000000000000000000000000/meta.json bucket=inmem level=debug msg="deleted file" file=00000000000000000000000000/deletion-mark.json bucket=inmem level=debug msg="deleted file" file=00000000010000000000000000/meta.json bucket=inmem level=debug msg="deleted file" file=00000000010000000000000000/deletion-mark.json bucket=inmem === RUN TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly/filesystem === PAUSE TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly/filesystem === CONT TestMultitenantCompactor_ShouldDoNothingOnNoUserBlocks --- PASS: TestMultitenantCompactor_ShouldDoNothingOnNoUserBlocks (0.05s) === CONT TestMultitenantCompactor_ShouldNotCompactBlocksForUsersMarkedForDeletion --- PASS: TestMultitenantCompactor_ShouldSkipCompactionForJobsNoMoreOwnedAfterPlanning (0.22s) === CONT TestMultitenantCompactor_ShouldCompactAllUsersOnShardingEnabledButOnlyOneInstanceRunning --- PASS: TestMultitenantCompactor_ShouldNotCompactBlocksForUsersMarkedForDeletion (0.25s) === CONT TestGroupCompactE2E === RUN TestGroupCompactE2E/inmem level=info msg="start sync of metas" level=info msg="start of GC" level=info msg="start of compactions" level=info msg="compaction iterations done" --- PASS: TestMultitenantCompactor_ShouldCompactAllUsersOnShardingEnabledButOnlyOneInstanceRunning (0.43s) === CONT TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForNoCompaction --- PASS: TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForNoCompaction (0.32s) === CONT TestMultitenantCompactor_ShouldIterateOverUsersAndRunCompaction --- PASS: TestMultitenantCompactor_ShouldIterateOverUsersAndRunCompaction (0.37s) === CONT TestMultitenantCompactor_ShouldStopCompactingTenantOnReachingMaxCompactionTime --- PASS: TestMultitenantCompactor_ShouldStopCompactingTenantOnReachingMaxCompactionTime (1.12s) === CONT TestMultitenantCompactor_ShouldIncrementCompactionErrorIfFailedToCompactASingleTenant --- PASS: TestMultitenantCompactor_ShouldIncrementCompactionErrorIfFailedToCompactASingleTenant (0.76s) === CONT TestSyncer_GarbageCollect_e2e/filesystem create 00000000000000000000000000 create 00000000010000000000000000 create 00000000020000000000000000 create 00000000030000000000000000 create 00000000040000000000000000 create 00000000050000000000000000 create 00000000060000000000000000 create 00000000070000000000000000 create 00000000080000000000000000 create 00000000090000000000000000 create 00000000340000000000000000 create 00000000680000000000000000 create 000000009C0000000000000000 create 00000000CG0000000000000000 create 00000000FM0000000000000000 === CONT TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly/filesystem create 00000000000000000000000000 --- PASS: TestSyncer_GarbageCollect_e2e (0.04s) --- PASS: TestSyncer_GarbageCollect_e2e/inmem (0.03s) --- PASS: TestSyncer_GarbageCollect_e2e/filesystem (0.08s) create 00000000010000000000000000 create 00000000340000000000000000 level=debug msg="deleted file" file=00000000000000000000000000/meta.json bucket="fs: /tmp/TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnlyfilesystem2992969310/001" level=debug msg="deleted file" file=00000000000000000000000000/deletion-mark.json bucket="fs: /tmp/TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnlyfilesystem2992969310/001" level=debug msg="deleted file" file=00000000010000000000000000/meta.json bucket="fs: /tmp/TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnlyfilesystem2992969310/001" level=debug msg="deleted file" file=00000000010000000000000000/deletion-mark.json bucket="fs: /tmp/TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnlyfilesystem2992969310/001" --- PASS: TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly (0.02s) --- PASS: TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly/inmem (0.02s) --- PASS: TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly/filesystem (0.02s) --- PASS: TestMultitenantCompactor_ShouldFailCompactionOnTimeout (5.12s) === CONT TestGroupCompactE2E/inmem bucket_compactor_e2e_test.go:476: Error Trace: /build/mimir/src/mimir/pkg/compactor/bucket_compactor_e2e_test.go:476 /build/mimir/src/mimir/pkg/compactor/bucket_compactor_e2e_test.go:452 /build/mimir/src/mimir/pkg/compactor/bucket_compactor_e2e_test.go:270 /build/mimir/src/mimir/pkg/compactor/bucket_compactor_e2e_test.go:592 Error: Received unexpected error: write block: 2 errors: populate block: writing block: add series: context deadline exceeded; context deadline exceeded Test: TestGroupCompactE2E/inmem --- FAIL: TestGroupCompactE2E (21.57s) --- FAIL: TestGroupCompactE2E/inmem (21.57s) --- PASS: TestMultitenantCompactor_ShouldCompactOnlyUsersOwnedByTheInstanceOnShardingEnabledAndMultipleInstancesRunning (23.99s) FAIL FAIL github.com/grafana/mimir/pkg/compactor 562.162s === RUN TestClient_WriteSeries === RUN TestClient_WriteSeries/write_series_in_a_single_batch === RUN TestClient_WriteSeries/write_series_in_multiple_batches === RUN TestClient_WriteSeries/request_failed_with_4xx_error === RUN TestClient_WriteSeries/request_failed_with_5xx_error --- PASS: TestClient_WriteSeries (0.02s) --- PASS: TestClient_WriteSeries/write_series_in_a_single_batch (0.01s) --- PASS: TestClient_WriteSeries/write_series_in_multiple_batches (0.00s) --- PASS: TestClient_WriteSeries/request_failed_with_4xx_error (0.00s) --- PASS: TestClient_WriteSeries/request_failed_with_5xx_error (0.00s) === RUN TestClient_QueryRange === RUN TestClient_QueryRange/results_cache_not_explicitly_disabled === RUN TestClient_QueryRange/results_cache_disabled --- PASS: TestClient_QueryRange (0.01s) --- PASS: TestClient_QueryRange/results_cache_not_explicitly_disabled (0.00s) --- PASS: TestClient_QueryRange/results_cache_disabled (0.00s) === RUN TestClient_Query === RUN TestClient_Query/results_cache_not_explicitly_disabled === RUN TestClient_Query/results_cache_disabled --- PASS: TestClient_Query (0.00s) --- PASS: TestClient_Query/results_cache_not_explicitly_disabled (0.00s) --- PASS: TestClient_Query/results_cache_disabled (0.00s) === RUN TestManager_PeriodicRun --- PASS: TestManager_PeriodicRun (0.05s) === RUN TestManager_SmokeTest === RUN TestManager_SmokeTest/successful_smoke_test === RUN TestManager_SmokeTest/failed_smoke_test --- PASS: TestManager_SmokeTest (0.00s) --- PASS: TestManager_SmokeTest/successful_smoke_test (0.00s) --- PASS: TestManager_SmokeTest/failed_smoke_test (0.00s) === RUN TestAlignTimestampToInterval --- PASS: TestAlignTimestampToInterval (0.00s) === RUN TestGetQueryStep === RUN TestGetQueryStep/should_return_write_interval_if_expected_number_of_samples_is_<_1000 === RUN TestGetQueryStep/should_align_step_to_write_interval_and_guarantee_no_more_than_1000_samples --- PASS: TestGetQueryStep (0.00s) --- PASS: TestGetQueryStep/should_return_write_interval_if_expected_number_of_samples_is_<_1000 (0.00s) --- PASS: TestGetQueryStep/should_align_step_to_write_interval_and_guarantee_no_more_than_1000_samples (0.00s) === RUN TestVerifySineWaveSamplesSum === RUN TestVerifySineWaveSamplesSum/should_return_error_if_there's_a_missing_sample === RUN TestVerifySineWaveSamplesSum/should_return_error_if_the_2nd_last_sample_has_an_unexpected_timestamp === RUN TestVerifySineWaveSamplesSum/should_return_no_error_if_all_samples_value_and_timestamp_match_the_expected_one_(1_series) === RUN TestVerifySineWaveSamplesSum/should_return_no_error_if_all_samples_value_and_timestamp_match_the_expected_one_(multiple_series) === RUN TestVerifySineWaveSamplesSum/should_return_error_if_there's_a_missing_series --- PASS: TestVerifySineWaveSamplesSum (0.00s) --- PASS: TestVerifySineWaveSamplesSum/should_return_error_if_there's_a_missing_sample (0.00s) --- PASS: TestVerifySineWaveSamplesSum/should_return_error_if_the_2nd_last_sample_has_an_unexpected_timestamp (0.00s) --- PASS: TestVerifySineWaveSamplesSum/should_return_no_error_if_all_samples_value_and_timestamp_match_the_expected_one_(1_series) (0.00s) --- PASS: TestVerifySineWaveSamplesSum/should_return_no_error_if_all_samples_value_and_timestamp_match_the_expected_one_(multiple_series) (0.00s) --- PASS: TestVerifySineWaveSamplesSum/should_return_error_if_there's_a_missing_series (0.00s) === RUN TestMinTime --- PASS: TestMinTime (0.00s) === RUN TestMaxTime --- PASS: TestMaxTime (0.00s) === RUN TestRandTime --- PASS: TestRandTime (0.00s) === RUN TestWriteReadSeriesTest_Run === RUN TestWriteReadSeriesTest_Run/should_write_series_with_current_timestamp_if_it's_already_aligned_to_write_interval === RUN TestWriteReadSeriesTest_Run/should_write_series_with_timestamp_aligned_to_write_interval === RUN TestWriteReadSeriesTest_Run/should_write_series_from_last_written_timestamp_until_now === RUN TestWriteReadSeriesTest_Run/should_stop_remote_writing_on_network_error === RUN TestWriteReadSeriesTest_Run/should_stop_remote_writing_on_5xx_error === RUN TestWriteReadSeriesTest_Run/should_keep_remote_writing_next_intervals_on_4xx_error === RUN TestWriteReadSeriesTest_Run/should_query_written_series,_compare_results_and_track_no_failure_if_results_match === RUN TestWriteReadSeriesTest_Run/should_query_written_series,_compare_results_and_track_failure_if_results_don't_match --- PASS: TestWriteReadSeriesTest_Run (4.09s) --- PASS: TestWriteReadSeriesTest_Run/should_write_series_with_current_timestamp_if_it's_already_aligned_to_write_interval (0.01s) --- PASS: TestWriteReadSeriesTest_Run/should_write_series_with_timestamp_aligned_to_write_interval (0.01s) --- PASS: TestWriteReadSeriesTest_Run/should_write_series_from_last_written_timestamp_until_now (2.03s) --- PASS: TestWriteReadSeriesTest_Run/should_stop_remote_writing_on_network_error (0.00s) --- PASS: TestWriteReadSeriesTest_Run/should_stop_remote_writing_on_5xx_error (0.00s) --- PASS: TestWriteReadSeriesTest_Run/should_keep_remote_writing_next_intervals_on_4xx_error (2.01s) --- PASS: TestWriteReadSeriesTest_Run/should_query_written_series,_compare_results_and_track_no_failure_if_results_match (0.01s) --- PASS: TestWriteReadSeriesTest_Run/should_query_written_series,_compare_results_and_track_failure_if_results_don't_match (0.01s) === RUN TestWriteReadSeriesTest_Init === RUN TestWriteReadSeriesTest_Init/no_previously_written_samples_found === RUN TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-2h,_-1m] === RUN TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-36h,_-1m] === RUN TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-36h,_-1m]_but_last_data_point_of_previous_24h_period_is_missing === RUN TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-24h,_-1m] === RUN TestWriteReadSeriesTest_Init/the_configured_query_max_age_is_>_24h === RUN TestWriteReadSeriesTest_Init/the_configured_query_max_age_is_<_24h === RUN TestWriteReadSeriesTest_Init/the_most_recent_previously_written_data_point_is_older_than_1h_ago === RUN TestWriteReadSeriesTest_Init/the_first_query_fails === RUN TestWriteReadSeriesTest_Init/a_subsequent_query_fails === RUN TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_in_the_middle_of_the_last_24h_period === RUN TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_in_the_middle_of_the_previous_24h_period === RUN TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_exactly_at_the_beginning_of_this_24h_period --- PASS: TestWriteReadSeriesTest_Init (0.16s) --- PASS: TestWriteReadSeriesTest_Init/no_previously_written_samples_found (0.00s) --- PASS: TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-2h,_-1m] (0.00s) --- PASS: TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-36h,_-1m] (0.01s) --- PASS: TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-36h,_-1m]_but_last_data_point_of_previous_24h_period_is_missing (0.01s) --- PASS: TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-24h,_-1m] (0.02s) --- PASS: TestWriteReadSeriesTest_Init/the_configured_query_max_age_is_>_24h (0.04s) --- PASS: TestWriteReadSeriesTest_Init/the_configured_query_max_age_is_<_24h (0.00s) --- PASS: TestWriteReadSeriesTest_Init/the_most_recent_previously_written_data_point_is_older_than_1h_ago (0.01s) --- PASS: TestWriteReadSeriesTest_Init/the_first_query_fails (0.00s) --- PASS: TestWriteReadSeriesTest_Init/a_subsequent_query_fails (0.01s) --- PASS: TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_in_the_middle_of_the_last_24h_period (0.01s) --- PASS: TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_in_the_middle_of_the_previous_24h_period (0.02s) --- PASS: TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_exactly_at_the_beginning_of_this_24h_period (0.02s) === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min/max_query_time_has_not_been_set_yet === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min/max_query_time_is_older_than_max_age === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_query_time_=_max_query_time === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_and_max_query_time_are_within_the_last_1h === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_and_max_query_time_are_within_the_last_2h === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_query_time_is_older_than_24h === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/max_query_time_is_older_than_24h_but_more_recent_than_max_query_age === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_query_time_is_older_than_24h_but_max_query_age_is_only_10m --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min/max_query_time_has_not_been_set_yet (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min/max_query_time_is_older_than_max_age (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_query_time_=_max_query_time (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_and_max_query_time_are_within_the_last_1h (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_and_max_query_time_are_within_the_last_2h (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_query_time_is_older_than_24h (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/max_query_time_is_older_than_24h_but_more_recent_than_max_query_age (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_query_time_is_older_than_24h_but_max_query_age_is_only_10m (0.00s) PASS ok github.com/grafana/mimir/pkg/continuoustest 4.493s === RUN TestConfig_Validate === RUN TestConfig_Validate/default_config_should_pass === RUN TestConfig_Validate/should_fail_if_the_default_shard_size_is_negative === RUN TestConfig_Validate/should_pass_if_the_default_shard_size_>=_0 --- PASS: TestConfig_Validate (0.02s) --- PASS: TestConfig_Validate/default_config_should_pass (0.00s) --- PASS: TestConfig_Validate/should_fail_if_the_default_shard_size_is_negative (0.01s) --- PASS: TestConfig_Validate/should_pass_if_the_default_shard_size_>=_0 (0.01s) === RUN TestDistributor_Push === RUN TestDistributor_Push/A_push_to_1_happy_ingesters_should_fail === RUN TestDistributor_Push/A_push_exceeding_burst_size_should_fail === RUN TestDistributor_Push/A_push_to_ingesters_with_an_old_sample_should_report_the_correct_metrics_with_no_metadata === RUN TestDistributor_Push/A_push_to_ingesters_without_samples_should_report_the_correct_metrics === RUN TestDistributor_Push/A_timed_out_push_should_fail === RUN TestDistributor_Push/A_push_to_3_happy_ingesters_should_succeed === RUN TestDistributor_Push/A_push_to_2_happy_ingesters_should_succeed === RUN TestDistributor_Push/A_push_to_ingesters_with_a_current_sample_should_report_the_correct_metrics_with_no_metadata === RUN TestDistributor_Push/A_push_of_no_samples_shouldn't_block_or_return_error,_even_if_ingesters_are_sad === RUN TestDistributor_Push/A_push_to_0_happy_ingesters_should_fail --- PASS: TestDistributor_Push (1.50s) --- PASS: TestDistributor_Push/A_push_to_1_happy_ingesters_should_fail (0.14s) --- PASS: TestDistributor_Push/A_push_exceeding_burst_size_should_fail (0.12s) --- PASS: TestDistributor_Push/A_push_to_ingesters_with_an_old_sample_should_report_the_correct_metrics_with_no_metadata (0.13s) --- PASS: TestDistributor_Push/A_push_to_ingesters_without_samples_should_report_the_correct_metrics (0.18s) --- PASS: TestDistributor_Push/A_timed_out_push_should_fail (0.15s) --- PASS: TestDistributor_Push/A_push_to_3_happy_ingesters_should_succeed (0.15s) --- PASS: TestDistributor_Push/A_push_to_2_happy_ingesters_should_succeed (0.15s) --- PASS: TestDistributor_Push/A_push_to_ingesters_with_a_current_sample_should_report_the_correct_metrics_with_no_metadata (0.21s) --- PASS: TestDistributor_Push/A_push_of_no_samples_shouldn't_block_or_return_error,_even_if_ingesters_are_sad (0.13s) --- PASS: TestDistributor_Push/A_push_to_0_happy_ingesters_should_fail (0.14s) === RUN TestDistributor_ContextCanceledRequest --- PASS: TestDistributor_ContextCanceledRequest (0.14s) === RUN TestDistributor_MetricsCleanup --- PASS: TestDistributor_MetricsCleanup (0.12s) === RUN TestDistributor_PushRequestRateLimiter === RUN TestDistributor_PushRequestRateLimiter/request_limit_should_be_evenly_shared_across_distributors === RUN TestDistributor_PushRequestRateLimiter/request_limit_is_disabled_when_set_to_0 === RUN TestDistributor_PushRequestRateLimiter/request_burst_should_set_to_each_distributor --- PASS: TestDistributor_PushRequestRateLimiter (0.44s) --- PASS: TestDistributor_PushRequestRateLimiter/request_limit_should_be_evenly_shared_across_distributors (0.17s) --- PASS: TestDistributor_PushRequestRateLimiter/request_limit_is_disabled_when_set_to_0 (0.14s) --- PASS: TestDistributor_PushRequestRateLimiter/request_burst_should_set_to_each_distributor (0.14s) === RUN TestDistributor_PushIngestionRateLimiter === RUN TestDistributor_PushIngestionRateLimiter/evenly_share_the_ingestion_limit_across_distributors === RUN TestDistributor_PushIngestionRateLimiter/for_each_distributor,_set_an_ingestion_burst_limit. --- PASS: TestDistributor_PushIngestionRateLimiter (0.27s) --- PASS: TestDistributor_PushIngestionRateLimiter/evenly_share_the_ingestion_limit_across_distributors (0.13s) --- PASS: TestDistributor_PushIngestionRateLimiter/for_each_distributor,_set_an_ingestion_burst_limit. (0.14s) === RUN TestDistributor_PushInstanceLimits === RUN TestDistributor_PushInstanceLimits/hits_rate_limit_on_first_request,_but_second_request_can_proceed === RUN TestDistributor_PushInstanceLimits/below_rate_limit_on_first_request,_but_hits_the_rate_limit_afterwards === RUN TestDistributor_PushInstanceLimits/below_inflight_size_limit === RUN TestDistributor_PushInstanceLimits/hits_inflight_size_limit === RUN TestDistributor_PushInstanceLimits/no_limits_limit === RUN TestDistributor_PushInstanceLimits/below_inflight_limit === RUN TestDistributor_PushInstanceLimits/hits_inflight_limit === RUN TestDistributor_PushInstanceLimits/below_ingestion_rate_limit --- PASS: TestDistributor_PushInstanceLimits (1.84s) --- PASS: TestDistributor_PushInstanceLimits/hits_rate_limit_on_first_request,_but_second_request_can_proceed (0.13s) --- PASS: TestDistributor_PushInstanceLimits/below_rate_limit_on_first_request,_but_hits_the_rate_limit_afterwards (0.84s) --- PASS: TestDistributor_PushInstanceLimits/below_inflight_size_limit (0.14s) --- PASS: TestDistributor_PushInstanceLimits/hits_inflight_size_limit (0.13s) --- PASS: TestDistributor_PushInstanceLimits/no_limits_limit (0.13s) --- PASS: TestDistributor_PushInstanceLimits/below_inflight_limit (0.19s) --- PASS: TestDistributor_PushInstanceLimits/hits_inflight_limit (0.13s) --- PASS: TestDistributor_PushInstanceLimits/below_ingestion_rate_limit (0.16s) === RUN TestDistributor_PushHAInstances === RUN TestDistributor_PushHAInstances/0 === RUN TestDistributor_PushHAInstances/1 === RUN TestDistributor_PushHAInstances/2 === RUN TestDistributor_PushHAInstances/3 --- PASS: TestDistributor_PushHAInstances (2.41s) --- PASS: TestDistributor_PushHAInstances/0 (2.02s) --- PASS: TestDistributor_PushHAInstances/1 (0.13s) --- PASS: TestDistributor_PushHAInstances/2 (0.12s) --- PASS: TestDistributor_PushHAInstances/3 (0.13s) === RUN TestDistributor_PushQuery === RUN TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=9) --- PASS: TestDistributor_PushQuery (0.18s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=1) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=0)) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=8) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=7) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=5) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=6) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=4) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=2) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=1) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=0) (0.14s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=8) (0.15s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) (0.21s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=9) (0.20s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=7) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=5) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=6) (0.18s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=4) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=3) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=2) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=9) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=8) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=7) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=6) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=4) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=5) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=3) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=2) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=0) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=9) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=7) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=8) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=6) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=5) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=4) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=3) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=1) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=2) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) (0.18s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=5)) (0.15s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=9) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=7) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=9) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=8) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=6) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=5) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=3) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=4) (0.19s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=2) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=9) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=8) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=7) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=5) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=3) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=2) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=4) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=8) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=7) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=5) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=6) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=4) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=0) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=8) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=6) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=7) (0.20s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=4) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=5) (0.24s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=3) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=2) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=0) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=1) (0.17s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) (0.18s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=8) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=7) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=5) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=4) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=6) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=3) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=0) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=8) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=7) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) (0.19s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=5) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=6) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=4) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=2) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=1) (0.14s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=0) (0.18s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=9) (0.21s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=8) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=7) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=6) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=5) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=3) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=4) (0.22s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=9) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=7) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=6) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=5) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=4) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=2) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=7) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=5) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=9) (0.22s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=2) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=4) (0.22s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=0) (0.21s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=2) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=6) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=7) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=8) (0.19s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=5) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=3) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=4) (0.20s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=2) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=1) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=0) (0.16s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=9) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=8) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=7) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=6) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=4) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=1) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=9) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=6) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=5) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=7) (0.18s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=4) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=3) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=2) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=9) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=8) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=6) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=7) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=4) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=3) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=9) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) (0.18s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=7) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=6) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=5) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=4) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=2) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=0) (0.14s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=7) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=9) (0.25s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=6) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=4) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=5) (0.22s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=3) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=2) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=0) (0.18s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) (0.16s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) (0.20s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=5)) (0.16s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=0)) (0.17s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=0)) (0.17s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=5)) (0.23s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=0)) (0.16s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=7) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=6) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=3) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=5) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=4) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=9) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=8) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=7) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=6) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=5) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=3) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=4) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=1) (0.14s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=8) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=7) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=4) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=3) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=2) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=0) (0.11s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=0)) (0.15s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=8) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=7) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=5) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=4) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=3) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=2) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=1) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=0) (0.15s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) (0.18s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=9) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=8) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=7) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=5) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=3) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=4) (0.20s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=1) (0.11s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=0) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=9) (0.18s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) (0.20s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=8) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=7) (0.20s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=5) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=6) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=4) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=3) (0.18s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=1) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=0) (0.17s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) (0.18s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=4) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=6) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=4) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=7) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=2) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=1) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=7) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=6) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=9) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=7) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=4) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=5) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=2) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=6) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=9) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=8) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=7) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=5)) (0.15s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=0)) (0.16s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=4) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=3) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=2) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=9) (0.11s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=7) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=6) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=4) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) (0.18s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=0) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=5)) (0.15s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=5)) (0.20s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=9) (0.12s) === RUN TestDistributor_QueryStream_ShouldReturnErrorIfMaxChunksPerQueryLimitIsReached --- PASS: TestDistributor_QueryStream_ShouldReturnErrorIfMaxChunksPerQueryLimitIsReached (0.14s) === RUN TestDistributor_QueryStream_ShouldReturnErrorIfMaxSeriesPerQueryLimitIsReached --- PASS: TestDistributor_QueryStream_ShouldReturnErrorIfMaxSeriesPerQueryLimitIsReached (0.21s) === RUN TestDistributor_QueryStream_ShouldReturnErrorIfMaxChunkBytesPerQueryLimitIsReached --- PASS: TestDistributor_QueryStream_ShouldReturnErrorIfMaxChunkBytesPerQueryLimitIsReached (0.12s) === RUN TestDistributor_Push_LabelRemoval --- PASS: TestDistributor_Push_LabelRemoval (0.37s) === RUN TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime === RUN TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_1 === RUN TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_1_and_dropped_label_due_to_config === RUN TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_1_and_dropped_HA_replica_label === RUN TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_2_with_value_1 === RUN TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_2 --- PASS: TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime (0.70s) --- PASS: TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_1 (0.12s) --- PASS: TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_1_and_dropped_label_due_to_config (0.13s) --- PASS: TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_1_and_dropped_HA_replica_label (0.11s) --- PASS: TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_2_with_value_1 (0.12s) --- PASS: TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_2 (0.22s) === RUN TestDistributor_Push_LabelNameValidation === RUN TestDistributor_Push_LabelNameValidation/label_name_validation_can_be_skipped_via_config === RUN TestDistributor_Push_LabelNameValidation/label_name_validation_can_be_skipped_via_WriteRequest_parameter === RUN TestDistributor_Push_LabelNameValidation/label_name_validation_is_on_by_default --- PASS: TestDistributor_Push_LabelNameValidation (0.44s) --- PASS: TestDistributor_Push_LabelNameValidation/label_name_validation_can_be_skipped_via_config (0.18s) --- PASS: TestDistributor_Push_LabelNameValidation/label_name_validation_can_be_skipped_via_WriteRequest_parameter (0.13s) --- PASS: TestDistributor_Push_LabelNameValidation/label_name_validation_is_on_by_default (0.13s) === RUN TestDistributor_Push_ExemplarValidation === RUN TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_no_labels === RUN TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_no_timestamp === RUN TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_too_long_labelset === RUN TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_too_many_series_labels === RUN TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_duplicate_series_labels === RUN TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_empty_series_label_name === RUN TestDistributor_Push_ExemplarValidation/valid_exemplar --- PASS: TestDistributor_Push_ExemplarValidation (0.97s) --- PASS: TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_no_labels (0.18s) --- PASS: TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_no_timestamp (0.13s) --- PASS: TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_too_long_labelset (0.13s) --- PASS: TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_too_many_series_labels (0.12s) --- PASS: TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_duplicate_series_labels (0.13s) --- PASS: TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_empty_series_label_name (0.13s) --- PASS: TestDistributor_Push_ExemplarValidation/valid_exemplar (0.16s) === RUN TestDistributor_ExemplarValidation === RUN TestDistributor_ExemplarValidation/one_old,_one_new,_same_series === RUN TestDistributor_ExemplarValidation/disable_exemplars === RUN TestDistributor_ExemplarValidation/valid_exemplars === RUN TestDistributor_ExemplarValidation/one_old,_one_new,_separate_series === RUN TestDistributor_ExemplarValidation/multi_exemplars --- PASS: TestDistributor_ExemplarValidation (0.65s) --- PASS: TestDistributor_ExemplarValidation/one_old,_one_new,_same_series (0.13s) --- PASS: TestDistributor_ExemplarValidation/disable_exemplars (0.17s) --- PASS: TestDistributor_ExemplarValidation/valid_exemplars (0.11s) --- PASS: TestDistributor_ExemplarValidation/one_old,_one_new,_separate_series (0.13s) --- PASS: TestDistributor_ExemplarValidation/multi_exemplars (0.11s) === RUN TestSlowQueries === RUN TestSlowQueries/0 === RUN TestSlowQueries/1 === RUN TestSlowQueries/2 === RUN TestSlowQueries/3 --- PASS: TestSlowQueries (0.89s) --- PASS: TestSlowQueries/0 (0.23s) --- PASS: TestSlowQueries/1 (0.22s) --- PASS: TestSlowQueries/2 (0.22s) --- PASS: TestSlowQueries/3 (0.23s) === RUN TestDistributor_MetricsForLabelMatchers === RUN TestDistributor_MetricsForLabelMatchers/should_filter_metrics_by_single_matcher === RUN TestDistributor_MetricsForLabelMatchers/should_filter_metrics_by_multiple_matchers === RUN TestDistributor_MetricsForLabelMatchers/should_return_all_matching_metrics_even_if_their_FastFingerprint_collide === RUN TestDistributor_MetricsForLabelMatchers/should_query_only_ingesters_belonging_to_tenant's_subring_if_shuffle_shard_size_is_set === RUN TestDistributor_MetricsForLabelMatchers/should_return_an_empty_response_if_no_metric_match --- PASS: TestDistributor_MetricsForLabelMatchers (0.60s) --- PASS: TestDistributor_MetricsForLabelMatchers/should_filter_metrics_by_single_matcher (0.12s) --- PASS: TestDistributor_MetricsForLabelMatchers/should_filter_metrics_by_multiple_matchers (0.12s) --- PASS: TestDistributor_MetricsForLabelMatchers/should_return_all_matching_metrics_even_if_their_FastFingerprint_collide (0.12s) --- PASS: TestDistributor_MetricsForLabelMatchers/should_query_only_ingesters_belonging_to_tenant's_subring_if_shuffle_shard_size_is_set (0.12s) --- PASS: TestDistributor_MetricsForLabelMatchers/should_return_an_empty_response_if_no_metric_match (0.12s) === RUN TestDistributor_LabelNames === RUN TestDistributor_LabelNames/should_return_an_empty_response_if_no_metric_match === RUN TestDistributor_LabelNames/should_filter_metrics_by_single_matcher === RUN TestDistributor_LabelNames/should_filter_metrics_by_multiple_matchers === RUN TestDistributor_LabelNames/should_query_only_ingesters_belonging_to_tenant's_subring_if_shuffle_sharding_is_enabled --- PASS: TestDistributor_LabelNames (0.47s) --- PASS: TestDistributor_LabelNames/should_return_an_empty_response_if_no_metric_match (0.12s) --- PASS: TestDistributor_LabelNames/should_filter_metrics_by_single_matcher (0.12s) --- PASS: TestDistributor_LabelNames/should_filter_metrics_by_multiple_matchers (0.12s) --- PASS: TestDistributor_LabelNames/should_query_only_ingesters_belonging_to_tenant's_subring_if_shuffle_sharding_is_enabled (0.11s) === RUN TestDistributor_MetricsMetadata === RUN TestDistributor_MetricsMetadata/should_query_all_ingesters_if_shuffle_sharding_is_enabled_but_shard_size_is_0 === RUN TestDistributor_MetricsMetadata/should_query_only_ingesters_belonging_to_tenant's_subring_if_shuffle_sharding_is_enabled --- PASS: TestDistributor_MetricsMetadata (0.24s) --- PASS: TestDistributor_MetricsMetadata/should_query_all_ingesters_if_shuffle_sharding_is_enabled_but_shard_size_is_0 (0.12s) --- PASS: TestDistributor_MetricsMetadata/should_query_only_ingesters_belonging_to_tenant's_subring_if_shuffle_sharding_is_enabled (0.12s) === RUN TestDistributor_LabelNamesAndValuesLimitTest === RUN TestDistributor_LabelNamesAndValuesLimitTest/expected_error_if_sizeLimit_is_reached === RUN TestDistributor_LabelNamesAndValuesLimitTest/expected_no_error_if_sizeLimit_is_not_reached --- PASS: TestDistributor_LabelNamesAndValuesLimitTest (0.23s) --- PASS: TestDistributor_LabelNamesAndValuesLimitTest/expected_error_if_sizeLimit_is_reached (0.12s) --- PASS: TestDistributor_LabelNamesAndValuesLimitTest/expected_no_error_if_sizeLimit_is_not_reached (0.12s) === RUN TestDistributor_LabelNamesAndValues === RUN TestDistributor_LabelNamesAndValues/should_group_values_of_labels_by_label_name_and_return_only_distinct_label_values === RUN TestDistributor_LabelNamesAndValues/should_return_the_results_if_zone_awareness_is_enabled_and_only_2_zones_return_the_results --- PASS: TestDistributor_LabelNamesAndValues (0.23s) --- PASS: TestDistributor_LabelNamesAndValues/should_group_values_of_labels_by_label_name_and_return_only_distinct_label_values (0.11s) --- PASS: TestDistributor_LabelNamesAndValues/should_return_the_results_if_zone_awareness_is_enabled_and_only_2_zones_return_the_results (0.12s) === RUN TestDistributor_LabelValuesCardinality_ExpectedAllIngestersResponsesToBeCompleted --- PASS: TestDistributor_LabelValuesCardinality_ExpectedAllIngestersResponsesToBeCompleted (8.00s) === RUN TestDistributor_LabelNamesAndValues_ExpectedAllPossibleLabelNamesAndValuesToBeReturned --- PASS: TestDistributor_LabelNamesAndValues_ExpectedAllPossibleLabelNamesAndValuesToBeReturned (13.37s) === RUN TestDistributor_IngestionIsControlledByForwarder === RUN TestDistributor_IngestionIsControlledByForwarder/do_ingest_with_only_samples === RUN TestDistributor_IngestionIsControlledByForwarder/don't_ingest_with_only_samples === RUN TestDistributor_IngestionIsControlledByForwarder/do_ingest_with_metadata === RUN TestDistributor_IngestionIsControlledByForwarder/don't_ingest_with_metadata === RUN TestDistributor_IngestionIsControlledByForwarder/do_ingest_with_exemplars === RUN TestDistributor_IngestionIsControlledByForwarder/don't_ingest_with_exemplars --- PASS: TestDistributor_IngestionIsControlledByForwarder (0.81s) --- PASS: TestDistributor_IngestionIsControlledByForwarder/do_ingest_with_only_samples (0.14s) --- PASS: TestDistributor_IngestionIsControlledByForwarder/don't_ingest_with_only_samples (0.12s) --- PASS: TestDistributor_IngestionIsControlledByForwarder/do_ingest_with_metadata (0.16s) --- PASS: TestDistributor_IngestionIsControlledByForwarder/don't_ingest_with_metadata (0.13s) --- PASS: TestDistributor_IngestionIsControlledByForwarder/do_ingest_with_exemplars (0.13s) --- PASS: TestDistributor_IngestionIsControlledByForwarder/don't_ingest_with_exemplars (0.13s) === RUN TestDistributor_LabelValuesCardinality === RUN TestDistributor_LabelValuesCardinality/should_return_an_empty_map_if_no_label_names === RUN TestDistributor_LabelValuesCardinality/should_return_a_map_with_the_label_values_and_series_occurrences_of_a_single_label_name === RUN TestDistributor_LabelValuesCardinality/should_return_a_map_with_the_label_values_and_series_occurrences_of_all_the_label_names --- PASS: TestDistributor_LabelValuesCardinality (0.35s) --- PASS: TestDistributor_LabelValuesCardinality/should_return_an_empty_map_if_no_label_names (0.11s) --- PASS: TestDistributor_LabelValuesCardinality/should_return_a_map_with_the_label_values_and_series_occurrences_of_a_single_label_name (0.11s) --- PASS: TestDistributor_LabelValuesCardinality/should_return_a_map_with_the_label_values_and_series_occurrences_of_all_the_label_names (0.13s) === RUN TestDistributor_LabelValuesCardinalityLimit === RUN TestDistributor_LabelValuesCardinalityLimit/should_return_a_httpgrpc_error_if_the_maximum_number_of_label_names_per_request_is_reached === RUN TestDistributor_LabelValuesCardinalityLimit/should_succeed_if_the_maximum_number_of_label_names_per_request_is_not_reached --- PASS: TestDistributor_LabelValuesCardinalityLimit (0.25s) --- PASS: TestDistributor_LabelValuesCardinalityLimit/should_return_a_httpgrpc_error_if_the_maximum_number_of_label_names_per_request_is_reached (0.13s) --- PASS: TestDistributor_LabelValuesCardinalityLimit/should_succeed_if_the_maximum_number_of_label_names_per_request_is_not_reached (0.12s) === RUN TestDistributor_LabelValuesCardinality_Concurrency === RUN TestDistributor_LabelValuesCardinality_Concurrency/should_fail_with_an_error_if_at_least_one_ingester's_LabelValuesCardinality_and/or_UserStats_operations_fails --- PASS: TestDistributor_LabelValuesCardinality_Concurrency (0.12s) --- PASS: TestDistributor_LabelValuesCardinality_Concurrency/should_fail_with_an_error_if_at_least_one_ingester's_LabelValuesCardinality_and/or_UserStats_operations_fails (0.12s) === RUN TestHaDedupeMiddleware === RUN TestHaDedupeMiddleware/no_changes_on_empty_request === RUN TestHaDedupeMiddleware/no_changes_if_accept_HA_samples_is_false === RUN TestHaDedupeMiddleware/remove_replica_label_with_HA_tracker_disabled === RUN TestHaDedupeMiddleware/do_nothing_without_user_in_context,_don't_even_call_next === RUN TestHaDedupeMiddleware/perform_HA_deduplication === RUN TestHaDedupeMiddleware/exceed_max_ha_clusters_limit --- PASS: TestHaDedupeMiddleware (5.39s) --- PASS: TestHaDedupeMiddleware/no_changes_on_empty_request (2.01s) --- PASS: TestHaDedupeMiddleware/no_changes_if_accept_HA_samples_is_false (2.01s) --- PASS: TestHaDedupeMiddleware/remove_replica_label_with_HA_tracker_disabled (0.11s) --- PASS: TestHaDedupeMiddleware/do_nothing_without_user_in_context,_don't_even_call_next (1.01s) --- PASS: TestHaDedupeMiddleware/perform_HA_deduplication (0.11s) --- PASS: TestHaDedupeMiddleware/exceed_max_ha_clusters_limit (0.13s) === RUN TestInstanceLimitsBeforeHaDedupe --- PASS: TestInstanceLimitsBeforeHaDedupe (0.13s) === RUN TestRelabelMiddleware === RUN TestRelabelMiddleware/do_nothing === RUN TestRelabelMiddleware/no_user_in_context === RUN TestRelabelMiddleware/apply_a_relabel_rule === RUN TestRelabelMiddleware/drop_two_out_of_three_labels === RUN TestRelabelMiddleware/drop_entire_series_if_they_have_no_labels --- PASS: TestRelabelMiddleware (0.62s) --- PASS: TestRelabelMiddleware/do_nothing (0.13s) --- PASS: TestRelabelMiddleware/no_user_in_context (0.12s) --- PASS: TestRelabelMiddleware/apply_a_relabel_rule (0.14s) --- PASS: TestRelabelMiddleware/drop_two_out_of_three_labels (0.11s) --- PASS: TestRelabelMiddleware/drop_entire_series_if_they_have_no_labels (0.11s) === RUN TestHaDedupeAndRelabelBeforeForwarding --- PASS: TestHaDedupeAndRelabelBeforeForwarding (0.13s) === RUN TestValidationBeforeForwarding --- PASS: TestValidationBeforeForwarding (2.01s) === RUN TestDistributorValidation === RUN TestDistributorValidation/0 === RUN TestDistributorValidation/1 === RUN TestDistributorValidation/2 === RUN TestDistributorValidation/3 === RUN TestDistributorValidation/4 === RUN TestDistributorValidation/5 --- PASS: TestDistributorValidation (0.74s) --- PASS: TestDistributorValidation/0 (0.14s) --- PASS: TestDistributorValidation/1 (0.11s) --- PASS: TestDistributorValidation/2 (0.12s) --- PASS: TestDistributorValidation/3 (0.12s) --- PASS: TestDistributorValidation/4 (0.13s) --- PASS: TestDistributorValidation/5 (0.11s) === RUN TestRemoveReplicaLabel --- PASS: TestRemoveReplicaLabel (0.00s) === RUN TestShardByAllLabelsReturnsWrongResultsForUnsortedLabels --- PASS: TestShardByAllLabelsReturnsWrongResultsForUnsortedLabels (0.00s) === RUN TestSortLabels --- PASS: TestSortLabels (0.00s) === RUN TestDistributor_Push_Relabel --- PASS: TestDistributor_Push_Relabel (0.26s) === RUN TestDistributor_MetricsWithRequestModifications === RUN TestDistributor_MetricsWithRequestModifications/No_modifications === RUN TestDistributor_MetricsWithRequestModifications/Drop_samples_via_relabel_rules === RUN TestDistributor_MetricsWithRequestModifications/Drop_samples_via_drop_label_rule === RUN TestDistributor_MetricsWithRequestModifications/Drop_requests_via_ha-dedupe === RUN TestDistributor_MetricsWithRequestModifications/Some_samples_get_forwarded === RUN TestDistributor_MetricsWithRequestModifications/Drop_examplars_via_limit === RUN TestDistributor_MetricsWithRequestModifications/Drop_half_of_samples_via_label_value_length_limit === RUN TestDistributor_MetricsWithRequestModifications/Drop_metadata_via_label_value_length_limit,_except_help --- PASS: TestDistributor_MetricsWithRequestModifications (2.39s) --- PASS: TestDistributor_MetricsWithRequestModifications/No_modifications (0.14s) --- PASS: TestDistributor_MetricsWithRequestModifications/Drop_samples_via_relabel_rules (0.12s) --- PASS: TestDistributor_MetricsWithRequestModifications/Drop_samples_via_drop_label_rule (0.12s) --- PASS: TestDistributor_MetricsWithRequestModifications/Drop_requests_via_ha-dedupe (0.12s) --- PASS: TestDistributor_MetricsWithRequestModifications/Some_samples_get_forwarded (0.21s) --- PASS: TestDistributor_MetricsWithRequestModifications/Drop_examplars_via_limit (0.13s) --- PASS: TestDistributor_MetricsWithRequestModifications/Drop_half_of_samples_via_label_value_length_limit (0.13s) --- PASS: TestDistributor_MetricsWithRequestModifications/Drop_metadata_via_label_value_length_limit,_except_help (0.13s) === RUN TestDistributor_CleanupIsDoneAfterLastIngesterReturns --- PASS: TestDistributor_CleanupIsDoneAfterLastIngesterReturns (0.12s) === RUN TestSeriesAreShardedToCorrectIngesters --- PASS: TestSeriesAreShardedToCorrectIngesters (0.30s) === RUN TestHATrackerConfig_Validate === PAUSE TestHATrackerConfig_Validate === RUN TestWatchPrefixAssignment --- PASS: TestWatchPrefixAssignment (0.00s) === RUN TestCheckReplicaOverwriteTimeout --- PASS: TestCheckReplicaOverwriteTimeout (1.00s) === RUN TestCheckReplicaMultiCluster --- PASS: TestCheckReplicaMultiCluster (1.00s) === RUN TestCheckReplicaMultiClusterTimeout --- PASS: TestCheckReplicaMultiClusterTimeout (1.00s) === RUN TestCheckReplicaUpdateTimeout --- PASS: TestCheckReplicaUpdateTimeout (1.00s) === RUN TestCheckReplicaMultiUser --- PASS: TestCheckReplicaMultiUser (1.00s) === RUN TestCheckReplicaUpdateTimeoutJitter === PAUSE TestCheckReplicaUpdateTimeoutJitter === RUN TestFindHALabels --- PASS: TestFindHALabels (0.00s) === RUN TestHATrackerConfig_ShouldCustomizePrefixDefaultValue --- PASS: TestHATrackerConfig_ShouldCustomizePrefixDefaultValue (0.00s) === RUN TestHAClustersLimit --- PASS: TestHAClustersLimit (0.06s) === RUN TestTooManyClustersError --- PASS: TestTooManyClustersError (0.00s) === RUN TestReplicasNotMatchError --- PASS: TestReplicasNotMatchError (0.00s) === RUN TestHATracker_MetricsCleanup --- PASS: TestHATracker_MetricsCleanup (0.00s) === RUN TestCheckReplicaCleanup --- PASS: TestCheckReplicaCleanup (2.00s) === RUN TestHealthyInstanceDelegate_OnRingInstanceHeartbeat === RUN TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/all_instances_healthy_and_active === RUN TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/all_instances_healthy_not_all_instances_active === RUN TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/some_instances_healthy_all_instances_active === RUN TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/some_instances_healthy_but_timeout_disabled_all_instances_active --- PASS: TestHealthyInstanceDelegate_OnRingInstanceHeartbeat (0.00s) --- PASS: TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/all_instances_healthy_and_active (0.00s) --- PASS: TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/all_instances_healthy_not_all_instances_active (0.00s) --- PASS: TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/some_instances_healthy_all_instances_active (0.00s) --- PASS: TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/some_instances_healthy_but_timeout_disabled_all_instances_active (0.00s) === RUN TestMergeSamplesIntoFirstDuplicates --- PASS: TestMergeSamplesIntoFirstDuplicates (0.00s) === RUN TestMergeSamplesIntoFirst --- PASS: TestMergeSamplesIntoFirst (0.00s) === RUN TestMergeSamplesIntoFirstNilA --- PASS: TestMergeSamplesIntoFirstNilA (0.00s) === RUN TestMergeSamplesIntoFirstNilB --- PASS: TestMergeSamplesIntoFirstNilB (0.00s) === RUN TestMergeExemplars === RUN TestMergeExemplars/test0 === RUN TestMergeExemplars/test1 === RUN TestMergeExemplars/test2 === RUN TestMergeExemplars/test3 === RUN TestMergeExemplars/test4 === RUN TestMergeExemplars/test5 === RUN TestMergeExemplars/test6 --- PASS: TestMergeExemplars (0.00s) --- PASS: TestMergeExemplars/test0 (0.00s) --- PASS: TestMergeExemplars/test1 (0.00s) --- PASS: TestMergeExemplars/test2 (0.00s) --- PASS: TestMergeExemplars/test3 (0.00s) --- PASS: TestMergeExemplars/test4 (0.00s) --- PASS: TestMergeExemplars/test5 (0.00s) --- PASS: TestMergeExemplars/test6 (0.00s) === RUN TestIngestionRateStrategy === RUN TestIngestionRateStrategy/rate_limiter_should_share_the_limit_across_the_number_of_distributors === RUN TestIngestionRateStrategy/infinite_rate_limiter_should_return_unlimited_settings --- PASS: TestIngestionRateStrategy (0.00s) --- PASS: TestIngestionRateStrategy/rate_limiter_should_share_the_limit_across_the_number_of_distributors (0.00s) --- PASS: TestIngestionRateStrategy/infinite_rate_limiter_should_return_unlimited_settings (0.00s) === CONT TestHATrackerConfig_Validate === CONT TestCheckReplicaUpdateTimeoutJitter === RUN TestCheckReplicaUpdateTimeoutJitter/should_not_refresh_the_replica_if_the_update_timeout_is_not_expired_yet_(with_jitter) === RUN TestCheckReplicaUpdateTimeoutJitter/should_refresh_the_replica_if_the_update_timeout_is_expired_(with_jitter) === RUN TestHATrackerConfig_Validate/should_fail_if_KV_backend_is_set_to_memberlist === RUN TestHATrackerConfig_Validate/should_pass_with_default_config === RUN TestHATrackerConfig_Validate/should_fail_if_max_update_timeout_jitter_is_negative === RUN TestHATrackerConfig_Validate/should_fail_if_failover_timeout_is_<_update_timeout_+_jitter_+_1_sec === RUN TestHATrackerConfig_Validate/should_pass_if_failover_timeout_is_>=_update_timeout_+_jitter_+_1_sec --- PASS: TestHATrackerConfig_Validate (0.01s) --- PASS: TestHATrackerConfig_Validate/should_fail_if_KV_backend_is_set_to_memberlist (0.00s) --- PASS: TestHATrackerConfig_Validate/should_pass_with_default_config (0.00s) --- PASS: TestHATrackerConfig_Validate/should_fail_if_max_update_timeout_jitter_is_negative (0.00s) --- PASS: TestHATrackerConfig_Validate/should_fail_if_failover_timeout_is_<_update_timeout_+_jitter_+_1_sec (0.00s) --- PASS: TestHATrackerConfig_Validate/should_pass_if_failover_timeout_is_>=_update_timeout_+_jitter_+_1_sec (0.00s) === RUN TestCheckReplicaUpdateTimeoutJitter/should_not_refresh_the_replica_if_the_update_timeout_is_not_expired_yet_(without_jitter) === RUN TestCheckReplicaUpdateTimeoutJitter/should_refresh_the_replica_if_the_update_timeout_is_expired_(without_jitter) --- PASS: TestCheckReplicaUpdateTimeoutJitter (2.01s) --- PASS: TestCheckReplicaUpdateTimeoutJitter/should_not_refresh_the_replica_if_the_update_timeout_is_not_expired_yet_(with_jitter) (0.00s) --- PASS: TestCheckReplicaUpdateTimeoutJitter/should_refresh_the_replica_if_the_update_timeout_is_expired_(with_jitter) (1.00s) --- PASS: TestCheckReplicaUpdateTimeoutJitter/should_not_refresh_the_replica_if_the_update_timeout_is_not_expired_yet_(without_jitter) (0.00s) --- PASS: TestCheckReplicaUpdateTimeoutJitter/should_refresh_the_replica_if_the_update_timeout_is_expired_(without_jitter) (1.00s) PASS ok github.com/grafana/mimir/pkg/distributor 73.002s ? github.com/grafana/mimir/pkg/distributor/distributorpb [no test files] === RUN TestForwardingSamplesSuccessfullyToSingleTarget --- PASS: TestForwardingSamplesSuccessfullyToSingleTarget (0.02s) === RUN TestForwardingOmitOldSamples === RUN TestForwardingOmitOldSamples/drop_nothing === RUN TestForwardingOmitOldSamples/drop_some_of_the_samples,_ingest_true === RUN TestForwardingOmitOldSamples/drop_some_of_the_samples,_ingest_false === RUN TestForwardingOmitOldSamples/split_one_sample_slice_in_the_middle === RUN TestForwardingOmitOldSamples/drop_all_of_the_samples --- PASS: TestForwardingOmitOldSamples (0.05s) --- PASS: TestForwardingOmitOldSamples/drop_nothing (0.02s) --- PASS: TestForwardingOmitOldSamples/drop_some_of_the_samples,_ingest_true (0.01s) --- PASS: TestForwardingOmitOldSamples/drop_some_of_the_samples,_ingest_false (0.01s) --- PASS: TestForwardingOmitOldSamples/split_one_sample_slice_in_the_middle (0.00s) --- PASS: TestForwardingOmitOldSamples/drop_all_of_the_samples (0.00s) === RUN TestForwardingEnsureThatPooledObjectsGetReturned === RUN TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_one_metric_and_don't_ingest_them === RUN TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_one_metric_and_ingest_them === RUN TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_and_don't_ingest_them === RUN TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_to_and_ingest_them === RUN TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_to_and_ingest_half_of_them_(1) === RUN TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_and_ingest_half_of_them_(2) === RUN TestForwardingEnsureThatPooledObjectsGetReturned/don't_forward_samples,_just_ingest_them --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned (0.05s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_one_metric_and_don't_ingest_them (0.00s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_one_metric_and_ingest_them (0.00s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_and_don't_ingest_them (0.01s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_to_and_ingest_them (0.01s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_to_and_ingest_half_of_them_(1) (0.02s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_and_ingest_half_of_them_(2) (0.01s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/don't_forward_samples,_just_ingest_them (0.00s) === RUN TestDropSamplesBefore === RUN TestDropSamplesBefore/no_samples === RUN TestDropSamplesBefore/all_samples_before_cutoff === RUN TestDropSamplesBefore/all_samples_after_cutoff === RUN TestDropSamplesBefore/cut_some_samples_off === RUN TestDropSamplesBefore/unsorted_sample_slice --- PASS: TestDropSamplesBefore (0.00s) --- PASS: TestDropSamplesBefore/no_samples (0.00s) --- PASS: TestDropSamplesBefore/all_samples_before_cutoff (0.00s) --- PASS: TestDropSamplesBefore/all_samples_after_cutoff (0.00s) --- PASS: TestDropSamplesBefore/cut_some_samples_off (0.00s) --- PASS: TestDropSamplesBefore/unsorted_sample_slice (0.00s) === RUN TestForwardingToHTTPGrpcTarget --- PASS: TestForwardingToHTTPGrpcTarget (0.01s) === RUN TestUsingPools --- PASS: TestUsingPools (0.00s) PASS ok github.com/grafana/mimir/pkg/distributor/forwarding 0.391s ? github.com/grafana/mimir/pkg/flusher [no test files] === RUN TestFrontend_RequestHostHeaderWhenDownstreamURLIsConfigured --- PASS: TestFrontend_RequestHostHeaderWhenDownstreamURLIsConfigured (0.10s) === RUN TestFrontend_LogsSlowQueriesFormValues --- PASS: TestFrontend_LogsSlowQueriesFormValues (0.03s) === RUN TestFrontend_ReturnsRequestBodyTooLargeError --- PASS: TestFrontend_ReturnsRequestBodyTooLargeError (0.05s) PASS ok github.com/grafana/mimir/pkg/frontend 0.745s === RUN TestRequest === RUN TestRequest/0 === RUN TestRequest/1 === RUN TestRequest/2 === RUN TestRequest/3 === RUN TestRequest/4 === RUN TestRequest/5 === RUN TestRequest/6 === RUN TestRequest/7 --- PASS: TestRequest (0.01s) --- PASS: TestRequest/0 (0.00s) --- PASS: TestRequest/1 (0.00s) --- PASS: TestRequest/2 (0.00s) --- PASS: TestRequest/3 (0.00s) --- PASS: TestRequest/4 (0.00s) --- PASS: TestRequest/5 (0.00s) --- PASS: TestRequest/6 (0.00s) --- PASS: TestRequest/7 (0.00s) === RUN TestDecodeFailedResponse === RUN TestDecodeFailedResponse/internal_error === RUN TestDecodeFailedResponse/too_many_requests === RUN TestDecodeFailedResponse/too_large_entry --- PASS: TestDecodeFailedResponse (0.00s) --- PASS: TestDecodeFailedResponse/internal_error (0.00s) --- PASS: TestDecodeFailedResponse/too_many_requests (0.00s) --- PASS: TestDecodeFailedResponse/too_large_entry (0.00s) === RUN TestResponseRoundtrip === RUN TestResponseRoundtrip/successful_string_response === RUN TestResponseRoundtrip/successful_scalar_response === RUN TestResponseRoundtrip/successful_instant_response === RUN TestResponseRoundtrip/successful_range_response === RUN TestResponseRoundtrip/successful_empty_matrix_response === RUN TestResponseRoundtrip/error_response --- PASS: TestResponseRoundtrip (0.01s) --- PASS: TestResponseRoundtrip/successful_string_response (0.00s) --- PASS: TestResponseRoundtrip/successful_scalar_response (0.00s) --- PASS: TestResponseRoundtrip/successful_instant_response (0.00s) --- PASS: TestResponseRoundtrip/successful_range_response (0.00s) --- PASS: TestResponseRoundtrip/successful_empty_matrix_response (0.00s) --- PASS: TestResponseRoundtrip/error_response (0.00s) === RUN TestMergeAPIResponses === RUN TestMergeAPIResponses/No_responses_shouldn't_panic_and_return_a_non-null_result_and_result_type. === RUN TestMergeAPIResponses/A_single_empty_response_shouldn't_panic. === RUN TestMergeAPIResponses/Multiple_empty_responses_shouldn't_panic. === RUN TestMergeAPIResponses/Basic_merging_of_two_responses. === RUN TestMergeAPIResponses/Merging_of_responses_when_labels_are_in_different_order. === RUN TestMergeAPIResponses/Merging_of_samples_where_there_is_single_overlap. === RUN TestMergeAPIResponses/Merging_of_samples_where_there_is_multiple_partial_overlaps. === RUN TestMergeAPIResponses/Merging_of_samples_where_there_is_complete_overlap. === RUN TestMergeAPIResponses/shouldn't_merge_unsuccessful_responses === RUN TestMergeAPIResponses/shouldn't_merge_nil_data === RUN TestMergeAPIResponses/shouldn't_merge_non-matrix_data --- PASS: TestMergeAPIResponses (0.00s) --- PASS: TestMergeAPIResponses/No_responses_shouldn't_panic_and_return_a_non-null_result_and_result_type. (0.00s) --- PASS: TestMergeAPIResponses/A_single_empty_response_shouldn't_panic. (0.00s) --- PASS: TestMergeAPIResponses/Multiple_empty_responses_shouldn't_panic. (0.00s) --- PASS: TestMergeAPIResponses/Basic_merging_of_two_responses. (0.00s) --- PASS: TestMergeAPIResponses/Merging_of_responses_when_labels_are_in_different_order. (0.00s) --- PASS: TestMergeAPIResponses/Merging_of_samples_where_there_is_single_overlap. (0.00s) --- PASS: TestMergeAPIResponses/Merging_of_samples_where_there_is_multiple_partial_overlaps. (0.00s) --- PASS: TestMergeAPIResponses/Merging_of_samples_where_there_is_complete_overlap. (0.00s) --- PASS: TestMergeAPIResponses/shouldn't_merge_unsuccessful_responses (0.00s) --- PASS: TestMergeAPIResponses/shouldn't_merge_nil_data (0.00s) --- PASS: TestMergeAPIResponses/shouldn't_merge_non-matrix_data (0.00s) === RUN Test_DecodeOptions === RUN Test_DecodeOptions/default === PAUSE Test_DecodeOptions/default === RUN Test_DecodeOptions/disable_cache === PAUSE Test_DecodeOptions/disable_cache === RUN Test_DecodeOptions/custom_sharding === PAUSE Test_DecodeOptions/custom_sharding === RUN Test_DecodeOptions/disable_sharding === PAUSE Test_DecodeOptions/disable_sharding === RUN Test_DecodeOptions/custom_instant_query_splitting === PAUSE Test_DecodeOptions/custom_instant_query_splitting === RUN Test_DecodeOptions/disable_instant_query_splitting === PAUSE Test_DecodeOptions/disable_instant_query_splitting === CONT Test_DecodeOptions/default === CONT Test_DecodeOptions/disable_instant_query_splitting === CONT Test_DecodeOptions/custom_instant_query_splitting === CONT Test_DecodeOptions/disable_sharding === CONT Test_DecodeOptions/custom_sharding === CONT Test_DecodeOptions/disable_cache --- PASS: Test_DecodeOptions (0.00s) --- PASS: Test_DecodeOptions/default (0.00s) --- PASS: Test_DecodeOptions/disable_instant_query_splitting (0.00s) --- PASS: Test_DecodeOptions/custom_instant_query_splitting (0.00s) --- PASS: Test_DecodeOptions/disable_sharding (0.00s) --- PASS: Test_DecodeOptions/custom_sharding (0.00s) --- PASS: Test_DecodeOptions/disable_cache (0.00s) === RUN TestLimitsMiddleware_MaxQueryLookback === RUN TestLimitsMiddleware_MaxQueryLookback/should_skip_executing_a_query_outside_the_allowed_time_range === RUN TestLimitsMiddleware_MaxQueryLookback/should_manipulate_a_query_where_maxQueryLookback_is_past_the_retention_period === RUN TestLimitsMiddleware_MaxQueryLookback/should_not_manipulate_time_range_if_max_lookback_is_disabled === RUN TestLimitsMiddleware_MaxQueryLookback/should_not_manipulate_time_range_for_a_query_on_short_time_range === RUN TestLimitsMiddleware_MaxQueryLookback/should_not_manipulate_a_query_on_large_time_range_close_to_the_limit === RUN TestLimitsMiddleware_MaxQueryLookback/should_manipulate_a_query_on_large_time_range_over_the_limit --- PASS: TestLimitsMiddleware_MaxQueryLookback (0.01s) --- PASS: TestLimitsMiddleware_MaxQueryLookback/should_skip_executing_a_query_outside_the_allowed_time_range (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLookback/should_manipulate_a_query_where_maxQueryLookback_is_past_the_retention_period (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLookback/should_not_manipulate_time_range_if_max_lookback_is_disabled (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLookback/should_not_manipulate_time_range_for_a_query_on_short_time_range (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLookback/should_not_manipulate_a_query_on_large_time_range_close_to_the_limit (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLookback/should_manipulate_a_query_on_large_time_range_over_the_limit (0.00s) === RUN TestLimitsMiddleware_MaxQueryLength === RUN TestLimitsMiddleware_MaxQueryLength/should_fail_on_a_query_on_large_time_range_over_the_limit,_ending_in_the_past === RUN TestLimitsMiddleware_MaxQueryLength/should_succeed_if_total_query_length_is_higher_than_query_length_limit === RUN TestLimitsMiddleware_MaxQueryLength/should_skip_validation_if_max_length_is_disabled === RUN TestLimitsMiddleware_MaxQueryLength/should_succeed_on_a_query_on_short_time_range,_ending_now === RUN TestLimitsMiddleware_MaxQueryLength/should_succeed_on_a_query_on_short_time_range,_ending_in_the_past === RUN TestLimitsMiddleware_MaxQueryLength/should_succeed_on_a_query_on_large_time_range_close_to_the_limit,_ending_now === RUN TestLimitsMiddleware_MaxQueryLength/should_fail_on_a_query_on_large_time_range_over_the_limit,_ending_now --- PASS: TestLimitsMiddleware_MaxQueryLength (0.01s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_fail_on_a_query_on_large_time_range_over_the_limit,_ending_in_the_past (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_succeed_if_total_query_length_is_higher_than_query_length_limit (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_skip_validation_if_max_length_is_disabled (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_succeed_on_a_query_on_short_time_range,_ending_now (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_succeed_on_a_query_on_short_time_range,_ending_in_the_past (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_succeed_on_a_query_on_large_time_range_close_to_the_limit,_ending_now (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_fail_on_a_query_on_large_time_range_over_the_limit,_ending_now (0.00s) === RUN TestLimitsMiddleware_CreationGracePeriod === RUN TestLimitsMiddleware_CreationGracePeriod/should_not_manipulate_time_range_if_creation_grace_period_is_disabled === RUN TestLimitsMiddleware_CreationGracePeriod/should_not_manipulate_time_range_for_a_query_in_now_+_creation_grace_period === RUN TestLimitsMiddleware_CreationGracePeriod/should_manipulate_time_range_for_a_query_over_now_+_creation_grace_period --- PASS: TestLimitsMiddleware_CreationGracePeriod (0.00s) --- PASS: TestLimitsMiddleware_CreationGracePeriod/should_not_manipulate_time_range_if_creation_grace_period_is_disabled (0.00s) --- PASS: TestLimitsMiddleware_CreationGracePeriod/should_not_manipulate_time_range_for_a_query_in_now_+_creation_grace_period (0.00s) --- PASS: TestLimitsMiddleware_CreationGracePeriod/should_manipulate_time_range_for_a_query_over_now_+_creation_grace_period (0.00s) === RUN TestLimitedRoundTripper_MaxQueryParallelism --- PASS: TestLimitedRoundTripper_MaxQueryParallelism (0.26s) === RUN TestLimitedRoundTripper_MaxQueryParallelismLateScheduling --- PASS: TestLimitedRoundTripper_MaxQueryParallelismLateScheduling (0.00s) === RUN TestLimitedRoundTripper_OriginalRequestContextCancellation --- PASS: TestLimitedRoundTripper_OriginalRequestContextCancellation (0.10s) === RUN TestQueryShardingCorrectness === RUN TestQueryShardingCorrectness/count()_grouping_'without' === PAUSE TestQueryShardingCorrectness/count()_grouping_'without' === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant === PAUSE TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr === PAUSE TestQueryShardingCorrectness/subquery_min_over_time_with_aggr === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_avg === PAUSE TestQueryShardingCorrectness/outer_subquery_on_top_of_avg === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1) === PAUSE TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1) === RUN TestQueryShardingCorrectness/min()_grouping_'by' === PAUSE TestQueryShardingCorrectness/min()_grouping_'by' === RUN TestQueryShardingCorrectness/scalar(single_metric) === PAUSE TestQueryShardingCorrectness/scalar(single_metric) === RUN TestQueryShardingCorrectness/vector(1) === PAUSE TestQueryShardingCorrectness/vector(1) === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1 === PAUSE TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1 === RUN TestQueryShardingCorrectness/avg()_grouping_'by' === PAUSE TestQueryShardingCorrectness/avg()_grouping_'by' === RUN TestQueryShardingCorrectness/or === PAUSE TestQueryShardingCorrectness/or === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order === PAUSE TestQueryShardingCorrectness/query_with_sort()_expects_specific_order === RUN TestQueryShardingCorrectness/stddev() === PAUSE TestQueryShardingCorrectness/stddev() === RUN TestQueryShardingCorrectness/month()_>=_1_and_month() === PAUSE TestQueryShardingCorrectness/month()_>=_1_and_month() === RUN TestQueryShardingCorrectness/avg()_no_grouping === PAUSE TestQueryShardingCorrectness/avg()_no_grouping === RUN TestQueryShardingCorrectness/sum()_grouping_'without' === PAUSE TestQueryShardingCorrectness/sum()_grouping_'without' === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without' === PAUSE TestQueryShardingCorrectness/histogram_quantile()_grouping_'without' === RUN TestQueryShardingCorrectness/topk() === PAUSE TestQueryShardingCorrectness/topk() === RUN TestQueryShardingCorrectness/sum({__name__!=""}) === PAUSE TestQueryShardingCorrectness/sum({__name__!=""}) === RUN TestQueryShardingCorrectness/sum()_no_grouping === PAUSE TestQueryShardingCorrectness/sum()_no_grouping === RUN TestQueryShardingCorrectness/max()_grouping_'without' === PAUSE TestQueryShardingCorrectness/max()_grouping_'without' === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children === PAUSE TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series === PAUSE TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series === RUN TestQueryShardingCorrectness/sum_of_subquery_min === PAUSE TestQueryShardingCorrectness/sum_of_subquery_min === RUN TestQueryShardingCorrectness/triple_subquery === PAUSE TestQueryShardingCorrectness/triple_subquery === RUN TestQueryShardingCorrectness/@_modifier_and_offset === PAUSE TestQueryShardingCorrectness/@_modifier_and_offset === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar === PAUSE TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar === PAUSE TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar === RUN TestQueryShardingCorrectness/sum(count()) === PAUSE TestQueryShardingCorrectness/sum(count()) === RUN TestQueryShardingCorrectness/sum(max_over_time()) === PAUSE TestQueryShardingCorrectness/sum(max_over_time()) === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter)) === PAUSE TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter)) === RUN TestQueryShardingCorrectness/scalar(aggregation) === PAUSE TestQueryShardingCorrectness/scalar(aggregation) === RUN TestQueryShardingCorrectness/bottomk() === PAUSE TestQueryShardingCorrectness/bottomk() === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""}) === PAUSE TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""}) === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping === PAUSE TestQueryShardingCorrectness/histogram_quantile()_no_grouping === RUN TestQueryShardingCorrectness/min()_no_grouping === PAUSE TestQueryShardingCorrectness/min()_no_grouping === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series === PAUSE TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series === RUN TestQueryShardingCorrectness/max()_grouping_'by' === PAUSE TestQueryShardingCorrectness/max()_grouping_'by' === RUN TestQueryShardingCorrectness/count()_grouping_'by' === PAUSE TestQueryShardingCorrectness/count()_grouping_'by' === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month() === PAUSE TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month() === RUN TestQueryShardingCorrectness/double_subquery_deriv === PAUSE TestQueryShardingCorrectness/double_subquery_deriv === RUN TestQueryShardingCorrectness/sum()_negative_offset === PAUSE TestQueryShardingCorrectness/sum()_negative_offset === RUN TestQueryShardingCorrectness/avg()_grouping_'without' === PAUSE TestQueryShardingCorrectness/avg()_grouping_'without' === RUN TestQueryShardingCorrectness/subquery_min === PAUSE TestQueryShardingCorrectness/subquery_min === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset === PAUSE TestQueryShardingCorrectness/@_modifier_and_negative_offset === RUN TestQueryShardingCorrectness/label_replace === PAUSE TestQueryShardingCorrectness/label_replace === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation === PAUSE TestQueryShardingCorrectness/histogram_quantile_without_aggregation === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1 === PAUSE TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1 === RUN TestQueryShardingCorrectness/sum()_grouping_'by' === PAUSE TestQueryShardingCorrectness/sum()_grouping_'by' === RUN TestQueryShardingCorrectness/@_modifier === PAUSE TestQueryShardingCorrectness/@_modifier === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_sum === PAUSE TestQueryShardingCorrectness/outer_subquery_on_top_of_sum === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1) === PAUSE TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1) === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2 === PAUSE TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation === PAUSE TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1) === PAUSE TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1) === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le === PAUSE TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate()) === PAUSE TestQueryShardingCorrectness/sum(rate())_>_avg(rate()) === RUN TestQueryShardingCorrectness/subquery_max === PAUSE TestQueryShardingCorrectness/subquery_max === RUN TestQueryShardingCorrectness/vector() === PAUSE TestQueryShardingCorrectness/vector() === RUN TestQueryShardingCorrectness/string_literal === PAUSE TestQueryShardingCorrectness/string_literal === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month() === PAUSE TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month() === RUN TestQueryShardingCorrectness/time() === PAUSE TestQueryShardingCorrectness/time() === RUN TestQueryShardingCorrectness/month(sum(metric_counter)) === PAUSE TestQueryShardingCorrectness/month(sum(metric_counter)) === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without' === PAUSE TestQueryShardingCorrectness/sum(rate())_grouping_'without' === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m])) === PAUSE TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m])) === RUN TestQueryShardingCorrectness/0_<_bool_1 === PAUSE TestQueryShardingCorrectness/0_<_bool_1 === RUN TestQueryShardingCorrectness/label_join === PAUSE TestQueryShardingCorrectness/label_join === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by' === PAUSE TestQueryShardingCorrectness/sum(rate())_grouping_'by' === RUN TestQueryShardingCorrectness/sum(avg_over_time()) === PAUSE TestQueryShardingCorrectness/sum(avg_over_time()) === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand === PAUSE TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1) === PAUSE TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1) === RUN TestQueryShardingCorrectness/min()_grouping_'without' === PAUSE TestQueryShardingCorrectness/min()_grouping_'without' === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping === PAUSE TestQueryShardingCorrectness/sum(rate())_no_grouping === RUN TestQueryShardingCorrectness/max()_no_grouping === PAUSE TestQueryShardingCorrectness/max()_no_grouping === RUN TestQueryShardingCorrectness/count()_no_grouping === PAUSE TestQueryShardingCorrectness/count()_no_grouping === RUN TestQueryShardingCorrectness/sum(min_over_time()) === PAUSE TestQueryShardingCorrectness/sum(min_over_time()) === RUN TestQueryShardingCorrectness/and === PAUSE TestQueryShardingCorrectness/and === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand === PAUSE TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand === PAUSE TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand === RUN TestQueryShardingCorrectness/sum()_offset === PAUSE TestQueryShardingCorrectness/sum()_offset === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2 === PAUSE TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2 === RUN TestQueryShardingCorrectness/nested_count() === PAUSE TestQueryShardingCorrectness/nested_count() === RUN TestQueryShardingCorrectness/stdvar() === PAUSE TestQueryShardingCorrectness/stdvar() === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by' === PAUSE TestQueryShardingCorrectness/histogram_quantile()_grouping_'by' === CONT TestQueryShardingCorrectness/sum(rate())_grouping_'without' === CONT TestQueryShardingCorrectness/histogram_quantile()_grouping_'by' === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === CONT TestQueryShardingCorrectness/histogram_quantile()_no_grouping === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === CONT TestQueryShardingCorrectness/count()_no_grouping === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/month(sum(metric_counter)) === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/time() === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month() === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/string_literal === RUN TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/vector() === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/subquery_max === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum(rate())_>_avg(rate()) === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1) === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1) === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/outer_subquery_on_top_of_sum === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/@_modifier === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum()_grouping_'by' === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/histogram_quantile_without_aggregation === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/label_replace === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/@_modifier_and_negative_offset === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/subquery_min === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/avg()_grouping_'without' === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/sum()_negative_offset === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/double_subquery_deriv === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month() === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/count()_grouping_'by' === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/max()_no_grouping === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/max()_grouping_'by' === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/sum(rate())_no_grouping === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/min()_grouping_'without' === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/min()_no_grouping === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum()_offset === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/stdvar() === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/nested_count() === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1) === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/sum(rate())_grouping_'by' === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/label_join === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/0_<_bool_1 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum(avg_over_time()) === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m])) === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest === CONT TestQueryShardingCorrectness/and === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum(min_over_time()) === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/max()_grouping_'without' === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""}) === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/@_modifier_and_offset === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/triple_subquery === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/bottomk() === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum_of_subquery_min === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/scalar(aggregation) === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter)) === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest === CONT TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/month()_>=_1_and_month() === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum()_no_grouping === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum({__name__!=""}) === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest === CONT TestQueryShardingCorrectness/topk() === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/histogram_quantile()_grouping_'without' === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/sum()_grouping_'without' === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/avg()_no_grouping === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum(count()) === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/avg()_grouping_'by' === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/stddev() === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest === CONT TestQueryShardingCorrectness/query_with_sort()_expects_specific_order === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/or === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/vector(1) === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/scalar(single_metric) === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum(max_over_time()) === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest === CONT TestQueryShardingCorrectness/min()_grouping_'by' === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest === CONT TestQueryShardingCorrectness/count()_grouping_'without' === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1) === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/outer_subquery_on_top_of_avg === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/subquery_min_over_time_with_aggr === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=16 --- PASS: TestQueryShardingCorrectness (0.76s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping (0.69s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest (0.24s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.05s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.05s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.07s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest (0.45s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.12s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.06s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.05s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.14s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping (2.75s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest (0.76s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.10s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.12s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.24s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.25s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest (1.99s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.46s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.32s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.31s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.59s) --- PASS: TestQueryShardingCorrectness/time() (0.33s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest (0.22s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.10s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.09s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.03s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest (0.11s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.03s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.05s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter)) (3.02s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest (1.18s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.17s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.15s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.33s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.45s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest (1.83s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.44s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.31s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.16s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.59s) --- PASS: TestQueryShardingCorrectness/string_literal (0.19s) --- PASS: TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest (0.19s) --- PASS: TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.04s) --- PASS: TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.06s) --- PASS: TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.03s) --- PASS: TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.03s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month() (0.83s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest (0.48s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.04s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.22s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.13s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.07s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest (0.34s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.14s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.03s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.06s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.04s) --- PASS: TestQueryShardingCorrectness/vector() (0.10s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest (0.06s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.02s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest (0.04s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without' (8.18s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest (1.04s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.22s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.17s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.19s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.30s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest (7.13s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (1.53s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (1.27s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (1.24s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (1.46s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate()) (7.24s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest (2.06s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.24s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.38s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.47s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.83s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest (5.17s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.71s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.76s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (1.00s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.64s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1) (0.49s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest (0.09s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.04s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.03s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest (0.40s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.04s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.03s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.22s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.05s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by' (18.31s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest (3.99s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.65s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.88s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.89s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.92s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest (14.33s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (1.89s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (1.48s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (1.44s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (2.24s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le (12.57s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest (3.69s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (1.53s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.48s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.44s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.44s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest (8.88s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (3.09s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.95s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.85s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (1.18s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2 (3.20s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest (0.71s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.12s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.12s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.12s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.30s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest (2.48s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.95s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.24s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.33s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.36s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_sum (0.33s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest (0.33s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.11s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.08s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.04s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.04s) --- PASS: TestQueryShardingCorrectness/@_modifier (2.41s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest (1.01s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.17s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.12s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.17s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.39s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest (1.40s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.21s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.26s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.31s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.41s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by' (1.62s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest (0.45s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.06s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.11s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.10s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.16s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest (1.17s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.14s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.23s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.34s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.26s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1 (0.50s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest (0.12s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.04s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.02s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest (0.38s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.12s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.07s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.06s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.05s) --- PASS: TestQueryShardingCorrectness/subquery_max (26.79s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest (4.70s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (1.01s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (1.32s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (1.15s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (1.08s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest (22.09s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (6.08s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (6.76s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (3.77s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (4.32s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation (4.47s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest (0.41s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.23s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.04s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.03s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.04s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest (4.07s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.65s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.76s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.81s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (1.32s) --- PASS: TestQueryShardingCorrectness/label_replace (1.26s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest (0.27s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.04s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.06s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.07s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.09s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest (0.98s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.22s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.26s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.16s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.18s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset (1.40s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest (0.52s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.09s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.08s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.12s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.16s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest (0.88s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.18s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.13s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.21s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.25s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without' (3.86s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest (0.80s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.06s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.09s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.20s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.42s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest (3.06s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.44s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.67s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.80s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.94s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset (1.31s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest (0.47s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.09s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.09s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.13s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.13s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest (0.83s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.21s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.12s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.14s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.18s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation (37.98s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest (2.28s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.47s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.31s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.39s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.73s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest (35.70s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (5.32s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (7.06s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (6.21s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (7.50s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month() (1.76s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest (0.48s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.05s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.10s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.13s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.15s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest (1.28s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.34s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.18s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.20s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.25s) --- PASS: TestQueryShardingCorrectness/subquery_min (20.17s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest (2.09s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.39s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.74s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.38s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.51s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest (18.08s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (4.31s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (4.26s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (4.80s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (4.39s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping (0.67s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest (0.27s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.05s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.03s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.04s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.13s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest (0.41s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.12s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.06s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.09s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.12s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by' (1.37s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest (0.42s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.04s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.08s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.10s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.16s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest (0.96s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.23s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.15s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.17s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.25s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by' (0.80s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest (0.27s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.05s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.06s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.11s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest (0.53s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.07s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.14s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.12s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.14s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping (2.37s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest (0.50s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.07s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.13s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.12s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.14s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest (1.87s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.64s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.25s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.26s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.37s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without' (1.21s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest (0.43s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.06s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.05s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.12s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.18s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest (0.77s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.16s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.11s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.24s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.17s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping (0.70s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest (0.26s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.04s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.15s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest (0.43s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.08s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.07s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.10s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.14s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series (4.89s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest (0.77s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.05s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.07s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.39s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.23s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest (4.13s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.92s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.83s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.85s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (1.04s) --- PASS: TestQueryShardingCorrectness/sum()_offset (1.53s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest (0.56s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.05s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.07s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.10s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.29s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest (0.96s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.12s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.31s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.15s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.24s) --- PASS: TestQueryShardingCorrectness/stdvar() (1.13s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest (0.26s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.04s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.04s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.03s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.04s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest (0.85s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.21s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.15s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.15s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.18s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv (22.43s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest (2.25s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.40s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.44s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.68s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.59s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest (20.18s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (3.68s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (5.42s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (4.60s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (5.96s) --- PASS: TestQueryShardingCorrectness/nested_count() (2.56s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest (0.57s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.08s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.10s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.10s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.19s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest (1.98s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.22s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.36s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.51s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.60s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1) (1.59s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest (0.57s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.05s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.10s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.11s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.26s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest (1.02s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.14s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.36s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.18s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.20s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2 (3.06s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest (0.77s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.10s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.08s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.16s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.29s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest (2.29s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.37s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.48s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.27s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.54s) --- PASS: TestQueryShardingCorrectness/label_join (1.79s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest (0.46s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.10s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.14s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.13s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest (1.33s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.12s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.39s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.25s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.33s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1 (0.28s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest (0.13s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.07s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.04s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest (0.13s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.06s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.02s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by' (2.90s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest (0.71s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.11s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.09s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.25s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.19s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest (2.19s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.46s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.38s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.35s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.50s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand (1.77s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest (0.65s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.06s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.14s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.15s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.19s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest (1.12s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.39s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.12s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.17s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.23s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand (1.66s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest (0.44s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.12s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.09s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.07s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.12s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest (1.22s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.29s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.16s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.20s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.35s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time()) (7.16s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest (1.02s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.23s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.15s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.18s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.28s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest (6.15s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (1.06s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (3.11s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.59s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.79s) --- PASS: TestQueryShardingCorrectness/and (3.53s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest (0.82s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.09s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.16s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.18s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.31s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest (2.71s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.34s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.52s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.59s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.94s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m])) (14.18s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest (5.02s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.37s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.50s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.81s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (2.67s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest (9.15s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (2.23s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (1.38s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (1.25s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (1.30s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time()) (4.84s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest (0.69s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.06s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.07s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.20s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.30s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest (4.14s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.98s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (1.02s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.76s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.72s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without' (0.86s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest (0.39s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.09s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.13s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.14s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest (0.47s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.09s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.09s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.10s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.13s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar (3.88s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest (0.44s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.04s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.07s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.15s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.12s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest (3.43s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.82s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.59s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.46s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.47s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""}) (10.15s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest (2.39s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.26s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.32s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.43s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.93s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest (7.74s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.99s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (1.10s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (2.91s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (1.29s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset (2.09s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest (0.88s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.13s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.14s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.22s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.25s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest (1.21s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.26s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.22s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.27s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.31s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar (28.71s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest (1.62s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.36s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.34s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.32s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.49s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest (27.08s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (6.25s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (9.47s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (5.67s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (4.32s) --- PASS: TestQueryShardingCorrectness/bottomk() (1.28s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest (0.40s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.10s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.12s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.09s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.05s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest (0.87s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.19s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.16s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.20s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.15s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1) (88.84s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest (2.68s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.58s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.45s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.78s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.69s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest (86.12s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (15.89s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (21.13s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (22.31s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (21.07s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation) (1.61s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest (0.66s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.07s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.08s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.10s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.23s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest (0.95s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.17s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.15s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.16s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.25s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min (2.59s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest (0.76s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.13s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.12s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.18s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.21s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest (1.82s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.40s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.29s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.32s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.40s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand (47.62s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest (4.40s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (2.40s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.57s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.47s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.40s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest (43.22s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (10.76s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (12.16s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (8.36s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (7.63s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children (3.22s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest (1.34s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.30s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.24s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.32s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.19s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest (1.87s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.48s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.32s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.34s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.41s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month() (0.21s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest (0.08s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.03s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest (0.13s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.06s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.03s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.02s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter)) (7.95s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest (0.88s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.08s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.14s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.16s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.40s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest (7.07s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (1.57s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.91s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (1.04s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.93s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping (2.52s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest (0.80s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.09s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.08s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.13s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.48s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest (1.72s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.18s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.14s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.85s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.21s) --- PASS: TestQueryShardingCorrectness/triple_subquery (26.82s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest (4.28s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.78s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.65s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (1.18s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (1.28s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest (22.54s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (5.12s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (4.59s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (5.50s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (6.57s) --- PASS: TestQueryShardingCorrectness/topk() (3.98s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest (0.39s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.04s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.04s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.07s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.17s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest (3.58s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.95s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.40s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.68s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.68s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""}) (8.07s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest (1.58s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.20s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.25s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.32s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.57s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest (6.48s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (1.97s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.75s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.81s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.92s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series (17.25s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest (0.86s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.13s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.22s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.17s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.29s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest (16.39s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (3.60s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (3.86s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (2.64s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (4.51s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without' (5.13s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest (1.43s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.38s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.37s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.29s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.22s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest (3.65s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.62s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.92s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.77s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.89s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping (2.72s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest (1.05s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.09s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.19s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.39s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.35s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest (1.67s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.15s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.38s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.46s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.51s) --- PASS: TestQueryShardingCorrectness/sum(count()) (2.61s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest (0.60s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.16s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.07s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.09s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.18s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest (2.01s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.20s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.28s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.45s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.52s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by' (2.83s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest (1.02s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.14s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.18s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.19s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.47s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest (1.81s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.37s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.41s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.30s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.52s) --- PASS: TestQueryShardingCorrectness/stddev() (1.64s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest (0.52s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.28s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.07s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.02s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.03s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest (1.12s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.11s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.37s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.25s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.19s) --- PASS: TestQueryShardingCorrectness/vector(1) (0.38s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest (0.17s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.06s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.03s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.03s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.05s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest (0.21s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.06s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.08s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.03s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.03s) --- PASS: TestQueryShardingCorrectness/or (2.47s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest (0.88s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.15s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.11s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.28s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.28s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest (1.58s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.48s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.19s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.20s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.47s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1 (2.48s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest (0.90s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.08s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.09s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.29s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.36s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest (1.57s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.19s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.36s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.35s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.37s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric) (0.74s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest (0.44s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.09s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.10s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.07s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.07s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest (0.30s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.09s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.03s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without' (14.03s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest (2.89s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.37s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.29s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.60s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (1.07s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest (11.14s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (2.66s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (1.66s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (1.24s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (1.27s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by' (1.62s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest (0.80s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.05s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.19s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.49s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest (0.83s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.24s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.19s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.14s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.24s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1) (2.04s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest (0.72s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.14s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.11s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.19s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.25s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest (1.31s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.24s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.24s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.31s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.27s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_avg (1.11s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest (1.11s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.25s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.40s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.26s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.07s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order (10.19s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest (2.34s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.28s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.29s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.56s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.79s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest (7.86s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (1.42s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (1.08s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.98s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (1.39s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without' (5.07s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest (0.59s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.05s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.07s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.21s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.23s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest (4.49s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.81s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.84s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (1.21s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (1.15s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time()) (6.47s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest (1.30s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.13s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.19s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.56s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.37s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest (5.16s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.78s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.87s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (1.23s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (1.56s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr (2.94s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest (1.84s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.84s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.55s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.07s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.16s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest (1.10s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.15s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.18s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.37s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.24s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant (21.91s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest (1.16s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.26s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.20s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.22s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.37s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest (20.76s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (5.85s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (3.79s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (4.34s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (4.65s) === RUN TestQueryshardingDeterminism --- PASS: TestQueryshardingDeterminism (0.87s) === RUN TestQuerySharding_FunctionCorrectness === RUN TestQuerySharding_FunctionCorrectness/(abs(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(abs(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(abs(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(abs(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(abs(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(avg_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(avg_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(avg_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(avg_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(avg_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(ceil(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(ceil(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(ceil(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(ceil(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(ceil(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(changes(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(changes(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(changes(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(changes(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(changes(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(count_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(count_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(count_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(count_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(count_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(days_in_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(days_in_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(days_in_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(days_in_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(days_in_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(day_of_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(day_of_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(day_of_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(day_of_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(day_of_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(day_of_week(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(day_of_week(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(day_of_week(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(day_of_week(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(day_of_week(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(day_of_year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(day_of_year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(day_of_year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(day_of_year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(day_of_year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(delta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(delta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(delta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(delta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(delta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(deriv(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(deriv(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(deriv(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(deriv(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(deriv(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(exp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(exp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(exp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(exp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(exp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(floor(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(floor(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(floor(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(floor(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(floor(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(hour(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(hour(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(hour(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(hour(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(hour(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(idelta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(idelta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(idelta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(idelta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(idelta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(increase(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(increase(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(increase(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(increase(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(increase(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(irate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(irate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(irate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(irate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(irate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(ln(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(ln(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(ln(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(ln(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(ln(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(log10(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(log10(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(log10(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(log10(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(log10(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(log2(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(log2(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(log2(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(log2(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(log2(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(max_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(max_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(max_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(max_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(max_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(min_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(min_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(min_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(min_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(min_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(minute(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(minute(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(minute(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(minute(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(minute(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(rate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(rate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(rate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(rate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(rate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(resets(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(resets(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(resets(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(resets(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(resets(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(sort(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(sort(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sort(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(sort(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(sort(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(sort_desc(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(sort_desc(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sort_desc(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(sort_desc(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(sort_desc(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(sqrt(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(sqrt(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sqrt(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(sqrt(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(sqrt(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(deg(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(deg(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(deg(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(deg(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(deg(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(asinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(asinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(asinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(asinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(asinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(rad(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(rad(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(rad(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(rad(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(rad(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(cosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(cosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(cosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(cosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(cosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(atan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(atan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(atan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(atan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(atan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(atanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(atanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(atanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(atanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(atanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(asin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(asin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(asin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(asin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(asin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(sinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(sinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(sinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(sinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(cos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(cos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(cos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(cos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(cos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(acosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(acosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(acosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(acosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(acosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(sin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(sin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(sin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(sin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(tanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(tanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(tanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(tanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(tanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(tan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(tan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(tan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(tan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(tan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(acos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(acos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(acos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(acos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(acos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(stddev_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(stddev_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(stddev_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(stddev_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(stddev_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(stdvar_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(stdvar_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(stdvar_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(stdvar_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(stdvar_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(sum_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(sum_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sum_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(sum_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(sum_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(last_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(last_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(last_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(last_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(last_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(present_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(present_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(present_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(present_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(present_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(quantile_over_time(0.5,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(quantile_over_time(0.5,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(quantile_over_time(0.5,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(quantile_over_time(0.5,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(quantile_over_time(0.5,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(quantile_over_time(0.99,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(quantile_over_time(0.99,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(quantile_over_time(0.99,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(quantile_over_time(0.99,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(quantile_over_time(0.99,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(timestamp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(timestamp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(timestamp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(timestamp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(timestamp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(sgn(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(sgn(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sgn(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(sgn(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(sgn(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(clamp(bar1{},5,10)) === RUN TestQuerySharding_FunctionCorrectness/sum(clamp(bar1{},5,10)) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(clamp(bar1{},5,10)) === RUN TestQuerySharding_FunctionCorrectness/count(clamp(bar1{},5,10)) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(clamp(bar1{},5,10)) === RUN TestQuerySharding_FunctionCorrectness/(clamp_max(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/sum(clamp_max(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(clamp_max(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/count(clamp_max(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(clamp_max(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/(clamp_min(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/sum(clamp_min(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(clamp_min(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/count(clamp_min(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(clamp_min(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/(predict_linear(bar1{}[1m],1)) === RUN TestQuerySharding_FunctionCorrectness/sum(predict_linear(bar1{}[1m],1)) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(predict_linear(bar1{}[1m],1)) === RUN TestQuerySharding_FunctionCorrectness/count(predict_linear(bar1{}[1m],1)) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(predict_linear(bar1{}[1m],1)) === RUN TestQuerySharding_FunctionCorrectness/(round(bar1{},20)) === RUN TestQuerySharding_FunctionCorrectness/sum(round(bar1{},20)) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(round(bar1{},20)) === RUN TestQuerySharding_FunctionCorrectness/count(round(bar1{},20)) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(round(bar1{},20)) === RUN TestQuerySharding_FunctionCorrectness/(holt_winters(bar1{}[1m],0.5,0.7)) === RUN TestQuerySharding_FunctionCorrectness/sum(holt_winters(bar1{}[1m],0.5,0.7)) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(holt_winters(bar1{}[1m],0.5,0.7)) === RUN TestQuerySharding_FunctionCorrectness/count(holt_winters(bar1{}[1m],0.5,0.7)) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(holt_winters(bar1{}[1m],0.5,0.7)) === RUN TestQuerySharding_FunctionCorrectness/(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) === RUN TestQuerySharding_FunctionCorrectness/sum(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) === RUN TestQuerySharding_FunctionCorrectness/count(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) === RUN TestQuerySharding_FunctionCorrectness/(label_join(bar1{},"fuzz",",","foo","bar")) === RUN TestQuerySharding_FunctionCorrectness/sum(label_join(bar1{},"fuzz",",","foo","bar")) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(label_join(bar1{},"fuzz",",","foo","bar")) === RUN TestQuerySharding_FunctionCorrectness/count(label_join(bar1{},"fuzz",",","foo","bar")) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(label_join(bar1{},"fuzz",",","foo","bar")) --- PASS: TestQuerySharding_FunctionCorrectness (10.04s) --- PASS: TestQuerySharding_FunctionCorrectness/(abs(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(abs(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(abs(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/count(abs(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(abs(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(avg_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(avg_over_time(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(avg_over_time(bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count(avg_over_time(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(avg_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(ceil(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(ceil(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(ceil(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(ceil(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(ceil(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/(changes(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(changes(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(changes(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(changes(bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(changes(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(count_over_time(bar1{}[1m])) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(count_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(count_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(count_over_time(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(count_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(days_in_month(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(days_in_month(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(days_in_month(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count(days_in_month(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(days_in_month(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(day_of_month(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(day_of_month(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(day_of_month(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count(day_of_month(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(day_of_month(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(day_of_week(bar1{})) (0.06s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(day_of_week(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(day_of_week(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count(day_of_week(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(day_of_week(bar1{})) (0.06s) --- PASS: TestQuerySharding_FunctionCorrectness/(day_of_year(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(day_of_year(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(day_of_year(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(day_of_year(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(day_of_year(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(delta(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(delta(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(delta(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(delta(bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(delta(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(deriv(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(deriv(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(deriv(bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count(deriv(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(deriv(bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(exp(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(exp(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(exp(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/count(exp(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(exp(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(floor(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(floor(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(floor(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/count(floor(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(floor(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(hour(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(hour(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(hour(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/count(hour(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(hour(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(idelta(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(idelta(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(idelta(bar1{}[1m])) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/count(idelta(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(idelta(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(increase(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(increase(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(increase(bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count(increase(bar1{}[1m])) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(increase(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(irate(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(irate(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(irate(bar1{}[1m])) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/count(irate(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(irate(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(ln(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(ln(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(ln(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(ln(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(ln(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(log10(bar1{})) (0.07s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(log10(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(log10(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count(log10(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(log10(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/(log2(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(log2(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(log2(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count(log2(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(log2(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(max_over_time(bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(max_over_time(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(max_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(max_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(max_over_time(bar1{}[1m])) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/(min_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(min_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(min_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(min_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(min_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(minute(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(minute(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(minute(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count(minute(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(minute(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(month(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(month(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(month(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(month(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(month(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/(rate(bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(rate(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(rate(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(rate(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(rate(bar1{}[1m])) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/(resets(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(resets(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(resets(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(resets(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(resets(bar1{}[1m])) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/(sort(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(sort(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sort(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(sort(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(sort(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(sort_desc(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(sort_desc(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sort_desc(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/count(sort_desc(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(sort_desc(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(sqrt(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(sqrt(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sqrt(bar1{})) (0.06s) --- PASS: TestQuerySharding_FunctionCorrectness/count(sqrt(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(sqrt(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(deg(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(deg(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(deg(bar1{})) (0.06s) --- PASS: TestQuerySharding_FunctionCorrectness/count(deg(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(deg(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(asinh(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(asinh(bar1{})) (0.06s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(asinh(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count(asinh(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(asinh(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(rad(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(rad(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(rad(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/count(rad(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(rad(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/(cosh(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(cosh(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(cosh(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/count(cosh(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(cosh(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(atan(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(atan(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(atan(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/count(atan(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(atan(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/(atanh(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(atanh(bar1{})) (0.07s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(atanh(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(atanh(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(atanh(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(asin(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(asin(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(asin(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(asin(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(asin(bar1{})) (0.07s) --- PASS: TestQuerySharding_FunctionCorrectness/(sinh(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(sinh(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sinh(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count(sinh(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(sinh(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/(cos(bar1{})) (0.06s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(cos(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(cos(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(cos(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(cos(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(acosh(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(acosh(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(acosh(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/count(acosh(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(acosh(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/(sin(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(sin(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sin(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/count(sin(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(sin(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(tanh(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(tanh(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(tanh(bar1{})) (0.07s) --- PASS: TestQuerySharding_FunctionCorrectness/count(tanh(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(tanh(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(tan(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(tan(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(tan(bar1{})) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/count(tan(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(tan(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(acos(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(acos(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(acos(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/count(acos(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(acos(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(stddev_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(stddev_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(stddev_over_time(bar1{}[1m])) (0.06s) --- PASS: TestQuerySharding_FunctionCorrectness/count(stddev_over_time(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(stddev_over_time(bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(stdvar_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(stdvar_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(stdvar_over_time(bar1{}[1m])) (0.07s) --- PASS: TestQuerySharding_FunctionCorrectness/count(stdvar_over_time(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(stdvar_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(sum_over_time(bar1{}[1m])) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(sum_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sum_over_time(bar1{}[1m])) (0.07s) --- PASS: TestQuerySharding_FunctionCorrectness/count(sum_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(sum_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(last_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(last_over_time(bar1{}[1m])) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(last_over_time(bar1{}[1m])) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/count(last_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(last_over_time(bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(present_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(present_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(present_over_time(bar1{}[1m])) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/count(present_over_time(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(present_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(quantile_over_time(0.5,bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(quantile_over_time(0.5,bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(quantile_over_time(0.5,bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count(quantile_over_time(0.5,bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(quantile_over_time(0.5,bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(quantile_over_time(0.99,bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(quantile_over_time(0.99,bar1{}[1m])) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(quantile_over_time(0.99,bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count(quantile_over_time(0.99,bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(quantile_over_time(0.99,bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(timestamp(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(timestamp(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(timestamp(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count(timestamp(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(timestamp(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(year(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(year(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(year(bar1{})) (0.06s) --- PASS: TestQuerySharding_FunctionCorrectness/count(year(bar1{})) (0.06s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(year(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(sgn(bar1{})) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(sgn(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sgn(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(sgn(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(sgn(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(clamp(bar1{},5,10)) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(clamp(bar1{},5,10)) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(clamp(bar1{},5,10)) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/count(clamp(bar1{},5,10)) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(clamp(bar1{},5,10)) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/(clamp_max(bar1{},5)) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(clamp_max(bar1{},5)) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(clamp_max(bar1{},5)) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count(clamp_max(bar1{},5)) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(clamp_max(bar1{},5)) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/(clamp_min(bar1{},5)) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(clamp_min(bar1{},5)) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(clamp_min(bar1{},5)) (0.07s) --- PASS: TestQuerySharding_FunctionCorrectness/count(clamp_min(bar1{},5)) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(clamp_min(bar1{},5)) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/(predict_linear(bar1{}[1m],1)) (0.06s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(predict_linear(bar1{}[1m],1)) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(predict_linear(bar1{}[1m],1)) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/count(predict_linear(bar1{}[1m],1)) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(predict_linear(bar1{}[1m],1)) (0.08s) --- PASS: TestQuerySharding_FunctionCorrectness/(round(bar1{},20)) (0.07s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(round(bar1{},20)) (0.06s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(round(bar1{},20)) (0.08s) --- PASS: TestQuerySharding_FunctionCorrectness/count(round(bar1{},20)) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(round(bar1{},20)) (0.06s) --- PASS: TestQuerySharding_FunctionCorrectness/(holt_winters(bar1{}[1m],0.5,0.7)) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(holt_winters(bar1{}[1m],0.5,0.7)) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(holt_winters(bar1{}[1m],0.5,0.7)) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/count(holt_winters(bar1{}[1m],0.5,0.7)) (0.05s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(holt_winters(bar1{}[1m],0.5,0.7)) (0.07s) --- PASS: TestQuerySharding_FunctionCorrectness/(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) (0.08s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) (0.08s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) (0.07s) --- PASS: TestQuerySharding_FunctionCorrectness/count(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) (0.06s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) (0.07s) --- PASS: TestQuerySharding_FunctionCorrectness/(label_join(bar1{},"fuzz",",","foo","bar")) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(label_join(bar1{},"fuzz",",","foo","bar")) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(label_join(bar1{},"fuzz",",","foo","bar")) (0.04s) --- PASS: TestQuerySharding_FunctionCorrectness/count(label_join(bar1{},"fuzz",",","foo","bar")) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(label_join(bar1{},"fuzz",",","foo","bar")) (0.05s) === RUN TestQuerySharding_ShouldFallbackToDownstreamHandlerOnMappingFailure --- PASS: TestQuerySharding_ShouldFallbackToDownstreamHandlerOnMappingFailure (0.01s) === RUN TestQuerySharding_ShouldSkipShardingViaOption --- PASS: TestQuerySharding_ShouldSkipShardingViaOption (0.01s) === RUN TestQuerySharding_ShouldOverrideShardingSizeViaOption --- PASS: TestQuerySharding_ShouldOverrideShardingSizeViaOption (0.31s) === RUN TestQuerySharding_ShouldSupportMaxShardedQueries === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/single_splitted_query,_query_has_1_shardable_leg === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/single_splitted_query,_query_has_many_shardable_legs === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_no_compactor_shards === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_3_compactor_shards === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_14_compactor_shards === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/query_sharding_is_disabled === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/query_is_not_shardable === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_1_shardable_leg === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_4_compactor_shards === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_10_compactor_shards === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_11_compactor_shards --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries (0.44s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/single_splitted_query,_query_has_1_shardable_leg (0.04s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/single_splitted_query,_query_has_many_shardable_legs (0.03s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs (0.01s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_no_compactor_shards (0.06s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_3_compactor_shards (0.03s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_14_compactor_shards (0.03s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/query_sharding_is_disabled (0.00s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/query_is_not_shardable (0.03s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_1_shardable_leg (0.02s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_4_compactor_shards (0.04s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_10_compactor_shards (0.06s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_11_compactor_shards (0.07s) === RUN TestQuerySharding_ShouldReturnErrorOnDownstreamHandlerFailure --- PASS: TestQuerySharding_ShouldReturnErrorOnDownstreamHandlerFailure (0.00s) === RUN TestQuerySharding_ShouldReturnErrorInCorrectFormat === RUN TestQuerySharding_ShouldReturnErrorInCorrectFormat/downstream_-_timeout === RUN TestQuerySharding_ShouldReturnErrorInCorrectFormat/sharding_-_sample_limit === RUN TestQuerySharding_ShouldReturnErrorInCorrectFormat/sharding_-_timeout === RUN TestQuerySharding_ShouldReturnErrorInCorrectFormat/downstream_-_storage_internal_error === RUN TestQuerySharding_ShouldReturnErrorInCorrectFormat/downstream_-_storage_prometheus_execution_error --- PASS: TestQuerySharding_ShouldReturnErrorInCorrectFormat (0.69s) --- PASS: TestQuerySharding_ShouldReturnErrorInCorrectFormat/downstream_-_timeout (0.34s) --- PASS: TestQuerySharding_ShouldReturnErrorInCorrectFormat/sharding_-_sample_limit (0.00s) --- PASS: TestQuerySharding_ShouldReturnErrorInCorrectFormat/sharding_-_timeout (0.34s) --- PASS: TestQuerySharding_ShouldReturnErrorInCorrectFormat/downstream_-_storage_internal_error (0.00s) --- PASS: TestQuerySharding_ShouldReturnErrorInCorrectFormat/downstream_-_storage_prometheus_execution_error (0.01s) === RUN TestQuerySharding_EngineErrorMapping --- PASS: TestQuerySharding_EngineErrorMapping (0.14s) === RUN TestQuerySharding_WrapMultipleTime --- PASS: TestQuerySharding_WrapMultipleTime (0.01s) === RUN TestPromqlResultToSampleStreams === RUN TestPromqlResultToSampleStreams/[0] === RUN TestPromqlResultToSampleStreams/[1] === RUN TestPromqlResultToSampleStreams/[2] === RUN TestPromqlResultToSampleStreams/[3] === RUN TestPromqlResultToSampleStreams/[4] --- PASS: TestPromqlResultToSampleStreams (0.00s) --- PASS: TestPromqlResultToSampleStreams/[0] (0.00s) --- PASS: TestPromqlResultToSampleStreams/[1] (0.00s) --- PASS: TestPromqlResultToSampleStreams/[2] (0.00s) --- PASS: TestPromqlResultToSampleStreams/[3] (0.00s) --- PASS: TestPromqlResultToSampleStreams/[4] (0.00s) === RUN TestGenLabelsCorrectness --- PASS: TestGenLabelsCorrectness (0.00s) === RUN TestGenLabelsSize --- PASS: TestGenLabelsSize (0.01s) === RUN TestNewMockShardedqueryable --- PASS: TestNewMockShardedqueryable (0.00s) === RUN TestResultsCacheConfig_Validate === RUN TestResultsCacheConfig_Validate/should_fail_with_unsupported_backend === RUN TestResultsCacheConfig_Validate/should_pass_with_memcached_backend === RUN TestResultsCacheConfig_Validate/should_pass_with_default_config === RUN TestResultsCacheConfig_Validate/should_fail_with_invalid_memcached_config --- PASS: TestResultsCacheConfig_Validate (0.00s) --- PASS: TestResultsCacheConfig_Validate/should_fail_with_unsupported_backend (0.00s) --- PASS: TestResultsCacheConfig_Validate/should_pass_with_memcached_backend (0.00s) --- PASS: TestResultsCacheConfig_Validate/should_pass_with_default_config (0.00s) --- PASS: TestResultsCacheConfig_Validate/should_fail_with_invalid_memcached_config (0.00s) === RUN TestIsRequestCachable === RUN TestIsRequestCachable/@_modifier_on_vector_selector,_before_end,_before_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_vector_selector,_after_end,_before_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_vector_selector,_before_end,_after_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_vector_selector,_after_end,_after_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_vector_selector_with_start()_before_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_vector_selector_with_end()_after_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_matrix_selector,_before_end,_before_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_matrix_selector,_after_end,_before_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_matrix_selector,_before_end,_after_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_matrix_selector,_after_end,_after_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_matrix_selector_with_start()_before_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_matrix_selector_with_end()_after_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_subqueries,_before_end,_before_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_subqueries,_after_end,_before_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_subqueries,_before_end,_after_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_subqueries,_after_end,_after_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_subqueries_with_start()_before_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_subqueries_with_end()_after_maxCacheTime === RUN TestIsRequestCachable/positive_offset_on_vector_selector === RUN TestIsRequestCachable/negative_offset_on_vector_selector === RUN TestIsRequestCachable/positive_offset_on_subqueries === RUN TestIsRequestCachable/negative_offset_on_subqueries === RUN TestIsRequestCachable/request_that_is_step_aligned === RUN TestIsRequestCachable/request_that_is_NOT_step_aligned,_with_cacheStepUnaligned=false === RUN TestIsRequestCachable/request_that_is_NOT_step_aligned --- PASS: TestIsRequestCachable (0.01s) --- PASS: TestIsRequestCachable/@_modifier_on_vector_selector,_before_end,_before_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_vector_selector,_after_end,_before_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_vector_selector,_before_end,_after_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_vector_selector,_after_end,_after_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_vector_selector_with_start()_before_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_vector_selector_with_end()_after_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_matrix_selector,_before_end,_before_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_matrix_selector,_after_end,_before_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_matrix_selector,_before_end,_after_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_matrix_selector,_after_end,_after_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_matrix_selector_with_start()_before_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_matrix_selector_with_end()_after_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_subqueries,_before_end,_before_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_subqueries,_after_end,_before_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_subqueries,_before_end,_after_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_subqueries,_after_end,_after_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_subqueries_with_start()_before_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_subqueries_with_end()_after_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/positive_offset_on_vector_selector (0.00s) --- PASS: TestIsRequestCachable/negative_offset_on_vector_selector (0.00s) --- PASS: TestIsRequestCachable/positive_offset_on_subqueries (0.00s) --- PASS: TestIsRequestCachable/negative_offset_on_subqueries (0.00s) --- PASS: TestIsRequestCachable/request_that_is_step_aligned (0.00s) --- PASS: TestIsRequestCachable/request_that_is_NOT_step_aligned,_with_cacheStepUnaligned=false (0.00s) --- PASS: TestIsRequestCachable/request_that_is_NOT_step_aligned (0.00s) === RUN TestIsResponseCachable === RUN TestIsResponseCachable/does_not_contain_the_cacheControl_header === RUN TestIsResponseCachable/does_contain_the_cacheControl_header_which_has_the_value === RUN TestIsResponseCachable/cacheControl_header_contains_extra_values_but_still_good === RUN TestIsResponseCachable/broken_response === RUN TestIsResponseCachable/nil_headers === RUN TestIsResponseCachable/had_cacheControl_header_but_no_values --- PASS: TestIsResponseCachable (0.00s) --- PASS: TestIsResponseCachable/does_not_contain_the_cacheControl_header (0.00s) --- PASS: TestIsResponseCachable/does_contain_the_cacheControl_header_which_has_the_value (0.00s) --- PASS: TestIsResponseCachable/cacheControl_header_contains_extra_values_but_still_good (0.00s) --- PASS: TestIsResponseCachable/broken_response (0.00s) --- PASS: TestIsResponseCachable/nil_headers (0.00s) --- PASS: TestIsResponseCachable/had_cacheControl_header_but_no_values (0.00s) === RUN TestPartitionCacheExtents === RUN TestPartitionCacheExtents/Test_a_complete_hit. === RUN TestPartitionCacheExtents/Test_with_a_complete_miss. === RUN TestPartitionCacheExtents/Test_a_partial_hit. === RUN TestPartitionCacheExtents/Test_multiple_partial_hits. === RUN TestPartitionCacheExtents/Partial_hits_with_tiny_gap. === RUN TestPartitionCacheExtents/Extent_is_outside_the_range_and_the_request_has_a_single_step_(same_start_and_end). === RUN TestPartitionCacheExtents/Test_when_hit_has_a_large_step_and_only_a_single_sample_extent. === RUN TestPartitionCacheExtents/Start_time_of_all_requests_must_have_the_same_offset_into_the_step. --- PASS: TestPartitionCacheExtents (0.00s) --- PASS: TestPartitionCacheExtents/Test_a_complete_hit. (0.00s) --- PASS: TestPartitionCacheExtents/Test_with_a_complete_miss. (0.00s) --- PASS: TestPartitionCacheExtents/Test_a_partial_hit. (0.00s) --- PASS: TestPartitionCacheExtents/Test_multiple_partial_hits. (0.00s) --- PASS: TestPartitionCacheExtents/Partial_hits_with_tiny_gap. (0.00s) --- PASS: TestPartitionCacheExtents/Extent_is_outside_the_range_and_the_request_has_a_single_step_(same_start_and_end). (0.00s) --- PASS: TestPartitionCacheExtents/Test_when_hit_has_a_large_step_and_only_a_single_sample_extent. (0.00s) --- PASS: TestPartitionCacheExtents/Start_time_of_all_requests_must_have_the_same_offset_into_the_step. (0.00s) === RUN TestConstSplitter_generateCacheKey === PAUSE TestConstSplitter_generateCacheKey === RUN TestRetry === RUN TestRetry/retry_failures === RUN TestRetry/don't_retry_400s === RUN TestRetry/retry_500s === RUN TestRetry/last_error --- PASS: TestRetry (0.01s) --- PASS: TestRetry/retry_failures (0.01s) --- PASS: TestRetry/don't_retry_400s (0.00s) --- PASS: TestRetry/retry_500s (0.00s) --- PASS: TestRetry/last_error (0.00s) === RUN Test_RetryMiddlewareCancel --- PASS: Test_RetryMiddlewareCancel (0.00s) === RUN TestRangeTripperware === RUN TestRangeTripperware/0 === RUN TestRangeTripperware/1 --- PASS: TestRangeTripperware (0.02s) --- PASS: TestRangeTripperware/0 (0.00s) --- PASS: TestRangeTripperware/1 (0.00s) === RUN TestInstantTripperware === RUN TestInstantTripperware/happy_case_roundtrip === RUN TestInstantTripperware/default_time_param --- PASS: TestInstantTripperware (0.03s) --- PASS: TestInstantTripperware/happy_case_roundtrip (0.01s) --- PASS: TestInstantTripperware/default_time_param (0.01s) === RUN TestTripperware_Metrics === RUN TestTripperware_Metrics/start/end_is_aligned_to_step === RUN TestTripperware_Metrics/start/end_is_not_aligned_to_step,_aligning_disabled === RUN TestTripperware_Metrics/start/end_is_not_aligned_to_step,_aligning_enabled --- PASS: TestTripperware_Metrics (0.05s) --- PASS: TestTripperware_Metrics/start/end_is_aligned_to_step (0.02s) --- PASS: TestTripperware_Metrics/start/end_is_not_aligned_to_step,_aligning_disabled (0.01s) --- PASS: TestTripperware_Metrics/start/end_is_not_aligned_to_step,_aligning_enabled (0.01s) === RUN TestShardedQuerier_Select === RUN TestShardedQuerier_Select/errors_non_embedded_query === RUN TestShardedQuerier_Select/replaces_query === RUN TestShardedQuerier_Select/propagates_response_error === RUN TestShardedQuerier_Select/returns_SeriesSet --- PASS: TestShardedQuerier_Select (0.00s) --- PASS: TestShardedQuerier_Select/errors_non_embedded_query (0.00s) --- PASS: TestShardedQuerier_Select/replaces_query (0.00s) --- PASS: TestShardedQuerier_Select/propagates_response_error (0.00s) --- PASS: TestShardedQuerier_Select/returns_SeriesSet (0.00s) === RUN TestShardedQuerier_Select_ShouldConcurrentlyRunEmbeddedQueries --- PASS: TestShardedQuerier_Select_ShouldConcurrentlyRunEmbeddedQueries (0.00s) === RUN TestShardedQueryable_GetResponseHeaders --- PASS: TestShardedQueryable_GetResponseHeaders (0.00s) === RUN TestNewSeriesSetFromEmbeddedQueriesResults === RUN TestNewSeriesSetFromEmbeddedQueriesResults/should_not_add_stale_markers_even_if_points_have_gaps_if_hints_is_not_passed === RUN TestNewSeriesSetFromEmbeddedQueriesResults/should_not_add_stale_markers_even_if_points_have_gaps_if_step_==_0 === RUN TestNewSeriesSetFromEmbeddedQueriesResults/should_add_a_stale_marker_at_the_end_even_if_if_input_samples_have_no_gaps === RUN TestNewSeriesSetFromEmbeddedQueriesResults/should_add_stale_markers_at_the_beginning_of_each_gap_and_one_at_the_end_of_the_series --- PASS: TestNewSeriesSetFromEmbeddedQueriesResults (0.01s) --- PASS: TestNewSeriesSetFromEmbeddedQueriesResults/should_not_add_stale_markers_even_if_points_have_gaps_if_hints_is_not_passed (0.01s) --- PASS: TestNewSeriesSetFromEmbeddedQueriesResults/should_not_add_stale_markers_even_if_points_have_gaps_if_step_==_0 (0.00s) --- PASS: TestNewSeriesSetFromEmbeddedQueriesResults/should_add_a_stale_marker_at_the_end_even_if_if_input_samples_have_no_gaps (0.00s) --- PASS: TestNewSeriesSetFromEmbeddedQueriesResults/should_add_stale_markers_at_the_beginning_of_each_gap_and_one_at_the_end_of_the_series (0.00s) === RUN TestResponseToSamples --- PASS: TestResponseToSamples (0.00s) === RUN TestSplitAndCacheMiddleware_SplitByInterval --- PASS: TestSplitAndCacheMiddleware_SplitByInterval (0.01s) === RUN TestSplitAndCacheMiddleware_ResultsCache --- PASS: TestSplitAndCacheMiddleware_ResultsCache (0.00s) === RUN TestSplitAndCacheMiddleware_ResultsCache_ShouldNotLookupCacheIfStepIsNotAligned --- PASS: TestSplitAndCacheMiddleware_ResultsCache_ShouldNotLookupCacheIfStepIsNotAligned (0.00s) === RUN TestSplitAndCacheMiddleware_ResultsCache_EnabledCachingOfStepUnalignedRequest --- PASS: TestSplitAndCacheMiddleware_ResultsCache_EnabledCachingOfStepUnalignedRequest (0.00s) === RUN TestSplitAndCacheMiddleware_ResultsCache_ShouldNotCacheRequestEarlierThanMaxCacheFreshness === RUN TestSplitAndCacheMiddleware_ResultsCache_ShouldNotCacheRequestEarlierThanMaxCacheFreshness/should_not_cache_a_response_if_query_time_range_is_earlier_than_max_cache_freshness === RUN TestSplitAndCacheMiddleware_ResultsCache_ShouldNotCacheRequestEarlierThanMaxCacheFreshness/should_cache_a_response_up_until_max_cache_freshness_time_ago --- PASS: TestSplitAndCacheMiddleware_ResultsCache_ShouldNotCacheRequestEarlierThanMaxCacheFreshness (0.01s) --- PASS: TestSplitAndCacheMiddleware_ResultsCache_ShouldNotCacheRequestEarlierThanMaxCacheFreshness/should_not_cache_a_response_if_query_time_range_is_earlier_than_max_cache_freshness (0.00s) --- PASS: TestSplitAndCacheMiddleware_ResultsCache_ShouldNotCacheRequestEarlierThanMaxCacheFreshness/should_cache_a_response_up_until_max_cache_freshness_time_ago (0.00s) === RUN TestSplitAndCacheMiddleware_ResultsCacheFuzzy split_and_cache_test.go:617: random generator seed: 1673979845641527840 === RUN TestSplitAndCacheMiddleware_ResultsCacheFuzzy/split_by_interval_disabled_(concurrency:_1) === PAUSE TestSplitAndCacheMiddleware_ResultsCacheFuzzy/split_by_interval_disabled_(concurrency:_1) === RUN TestSplitAndCacheMiddleware_ResultsCacheFuzzy/split_by_interval_disabled_(concurrency:_10) === PAUSE TestSplitAndCacheMiddleware_ResultsCacheFuzzy/split_by_interval_disabled_(concurrency:_10) === RUN TestSplitAndCacheMiddleware_ResultsCacheFuzzy/results_cache_disabled_(concurrency:_1) === PAUSE TestSplitAndCacheMiddleware_ResultsCacheFuzzy/results_cache_disabled_(concurrency:_1) === RUN TestSplitAndCacheMiddleware_ResultsCacheFuzzy/results_cache_disabled_(concurrency:_10) === PAUSE TestSplitAndCacheMiddleware_ResultsCacheFuzzy/results_cache_disabled_(concurrency:_10) === RUN TestSplitAndCacheMiddleware_ResultsCacheFuzzy/both_split_by_interval_and_results_cache_disabled_(concurrency:_1) === PAUSE TestSplitAndCacheMiddleware_ResultsCacheFuzzy/both_split_by_interval_and_results_cache_disabled_(concurrency:_1) === RUN TestSplitAndCacheMiddleware_ResultsCacheFuzzy/both_split_by_interval_and_results_cache_disabled_(concurrency:_10) === PAUSE TestSplitAndCacheMiddleware_ResultsCacheFuzzy/both_split_by_interval_and_results_cache_disabled_(concurrency:_10) === RUN TestSplitAndCacheMiddleware_ResultsCacheFuzzy/default_config_(concurrency:_1) === PAUSE TestSplitAndCacheMiddleware_ResultsCacheFuzzy/default_config_(concurrency:_1) === RUN TestSplitAndCacheMiddleware_ResultsCacheFuzzy/default_config_(concurrency:_10) === PAUSE TestSplitAndCacheMiddleware_ResultsCacheFuzzy/default_config_(concurrency:_10) === RUN TestSplitAndCacheMiddleware_ResultsCacheFuzzy/reduced_query_parallelism_(concurrency:_1) === PAUSE TestSplitAndCacheMiddleware_ResultsCacheFuzzy/reduced_query_parallelism_(concurrency:_1) === RUN TestSplitAndCacheMiddleware_ResultsCacheFuzzy/reduced_query_parallelism_(concurrency:_10) === PAUSE TestSplitAndCacheMiddleware_ResultsCacheFuzzy/reduced_query_parallelism_(concurrency:_10) === RUN TestSplitAndCacheMiddleware_ResultsCacheFuzzy/cache_unaligned_requests_(concurrency:_1) === PAUSE TestSplitAndCacheMiddleware_ResultsCacheFuzzy/cache_unaligned_requests_(concurrency:_1) === RUN TestSplitAndCacheMiddleware_ResultsCacheFuzzy/cache_unaligned_requests_(concurrency:_10) === PAUSE TestSplitAndCacheMiddleware_ResultsCacheFuzzy/cache_unaligned_requests_(concurrency:_10) === RUN TestSplitAndCacheMiddleware_ResultsCacheFuzzy/increased_max_cache_freshness_(concurrency:_1) === PAUSE TestSplitAndCacheMiddleware_ResultsCacheFuzzy/increased_max_cache_freshness_(concurrency:_1) === RUN TestSplitAndCacheMiddleware_ResultsCacheFuzzy/increased_max_cache_freshness_(concurrency:_10) === PAUSE TestSplitAndCacheMiddleware_ResultsCacheFuzzy/increased_max_cache_freshness_(concurrency:_10) === CONT TestSplitAndCacheMiddleware_ResultsCacheFuzzy/split_by_interval_disabled_(concurrency:_1) === CONT TestSplitAndCacheMiddleware_ResultsCacheFuzzy/default_config_(concurrency:_10) === CONT TestSplitAndCacheMiddleware_ResultsCacheFuzzy/increased_max_cache_freshness_(concurrency:_10) === CONT TestSplitAndCacheMiddleware_ResultsCacheFuzzy/increased_max_cache_freshness_(concurrency:_1) === CONT TestSplitAndCacheMiddleware_ResultsCacheFuzzy/cache_unaligned_requests_(concurrency:_10) === CONT TestSplitAndCacheMiddleware_ResultsCacheFuzzy/cache_unaligned_requests_(concurrency:_1) === CONT TestSplitAndCacheMiddleware_ResultsCacheFuzzy/reduced_query_parallelism_(concurrency:_10) === CONT TestSplitAndCacheMiddleware_ResultsCacheFuzzy/reduced_query_parallelism_(concurrency:_1) === CONT TestSplitAndCacheMiddleware_ResultsCacheFuzzy/results_cache_disabled_(concurrency:_1) === CONT TestSplitAndCacheMiddleware_ResultsCacheFuzzy/results_cache_disabled_(concurrency:_10) === CONT TestSplitAndCacheMiddleware_ResultsCacheFuzzy/split_by_interval_disabled_(concurrency:_10) === CONT TestSplitAndCacheMiddleware_ResultsCacheFuzzy/default_config_(concurrency:_1) === CONT TestSplitAndCacheMiddleware_ResultsCacheFuzzy/both_split_by_interval_and_results_cache_disabled_(concurrency:_1) === CONT TestSplitAndCacheMiddleware_ResultsCacheFuzzy/both_split_by_interval_and_results_cache_disabled_(concurrency:_10) --- PASS: TestSplitAndCacheMiddleware_ResultsCacheFuzzy (15.12s) --- PASS: TestSplitAndCacheMiddleware_ResultsCacheFuzzy/default_config_(concurrency:_10) (23.11s) --- PASS: TestSplitAndCacheMiddleware_ResultsCacheFuzzy/increased_max_cache_freshness_(concurrency:_10) (23.13s) --- PASS: TestSplitAndCacheMiddleware_ResultsCacheFuzzy/cache_unaligned_requests_(concurrency:_10) (12.58s) --- PASS: TestSplitAndCacheMiddleware_ResultsCacheFuzzy/reduced_query_parallelism_(concurrency:_10) (12.61s) --- PASS: TestSplitAndCacheMiddleware_ResultsCacheFuzzy/increased_max_cache_freshness_(concurrency:_1) (66.74s) --- PASS: TestSplitAndCacheMiddleware_ResultsCacheFuzzy/cache_unaligned_requests_(concurrency:_1) (48.14s) --- PASS: TestSplitAndCacheMiddleware_ResultsCacheFuzzy/results_cache_disabled_(concurrency:_10) (12.12s) --- PASS: TestSplitAndCacheMiddleware_ResultsCacheFuzzy/reduced_query_parallelism_(concurrency:_1) (44.99s) --- PASS: TestSplitAndCacheMiddleware_ResultsCacheFuzzy/split_by_interval_disabled_(concurrency:_1) (99.80s) --- PASS: TestSplitAndCacheMiddleware_ResultsCacheFuzzy/split_by_interval_disabled_(concurrency:_10) (22.10s) --- PASS: TestSplitAndCacheMiddleware_ResultsCacheFuzzy/results_cache_disabled_(concurrency:_1) (54.12s) --- PASS: TestSplitAndCacheMiddleware_ResultsCacheFuzzy/both_split_by_interval_and_results_cache_disabled_(concurrency:_10) (16.78s) --- PASS: TestSplitAndCacheMiddleware_ResultsCacheFuzzy/default_config_(concurrency:_1) (34.19s) --- PASS: TestSplitAndCacheMiddleware_ResultsCacheFuzzy/both_split_by_interval_and_results_cache_disabled_(concurrency:_1) (34.76s) === RUN TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases === RUN TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_not_drop_tiny_extent_that_partially_center-overlaps_with_tiny_request === RUN TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_not_drop_tiny_extent_that_partially_left-overlaps_with_tiny_request === RUN TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_not_drop_tiny_extent_that_partially_right-overlaps_with_tiny_request === RUN TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_left-extend_extent_if_request_starts_earlier_than_extent_in_cache === RUN TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_right-extend_extent_if_request_ends_later_than_extent_in_cache === RUN TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_not_throw_error_if_complete-overlapped_smaller_Extent_is_erroneous === RUN TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_replace_tiny_extents_that_are_cover_by_bigger_request === RUN TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_not_drop_tiny_extent_that_completely_overlaps_with_tiny_request === RUN TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_merge_fragmented_extents_if_request_fills_the_hole === RUN TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_drop_tiny_extent_that_overlaps_with_non-tiny_request_only --- PASS: TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases (0.01s) --- PASS: TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_not_drop_tiny_extent_that_partially_center-overlaps_with_tiny_request (0.00s) --- PASS: TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_not_drop_tiny_extent_that_partially_left-overlaps_with_tiny_request (0.00s) --- PASS: TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_not_drop_tiny_extent_that_partially_right-overlaps_with_tiny_request (0.00s) --- PASS: TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_left-extend_extent_if_request_starts_earlier_than_extent_in_cache (0.00s) --- PASS: TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_right-extend_extent_if_request_ends_later_than_extent_in_cache (0.00s) --- PASS: TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_not_throw_error_if_complete-overlapped_smaller_Extent_is_erroneous (0.00s) --- PASS: TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_replace_tiny_extents_that_are_cover_by_bigger_request (0.00s) --- PASS: TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_not_drop_tiny_extent_that_completely_overlaps_with_tiny_request (0.00s) --- PASS: TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_merge_fragmented_extents_if_request_fills_the_hole (0.00s) --- PASS: TestSplitAndCacheMiddleware_ResultsCache_ExtentsEdgeCases/Should_drop_tiny_extent_that_overlaps_with_non-tiny_request_only (0.00s) === RUN TestSplitAndCacheMiddleware_StoreAndFetchCacheExtents === RUN TestSplitAndCacheMiddleware_StoreAndFetchCacheExtents/fetchCacheExtents()_should_return_a_slice_with_the_same_number_of_input_keys_but_empty_extents_on_cache_miss === RUN TestSplitAndCacheMiddleware_StoreAndFetchCacheExtents/fetchCacheExtents()_should_return_a_slice_with_the_same_number_of_input_keys_and_some_extends_filled_up_on_partial_cache_hit === RUN TestSplitAndCacheMiddleware_StoreAndFetchCacheExtents/fetchCacheExtents()_should_not_return_an_extent_if_its_key_doesn't_match_the_requested_one_(hash_collision) --- PASS: TestSplitAndCacheMiddleware_StoreAndFetchCacheExtents (0.00s) --- PASS: TestSplitAndCacheMiddleware_StoreAndFetchCacheExtents/fetchCacheExtents()_should_return_a_slice_with_the_same_number_of_input_keys_but_empty_extents_on_cache_miss (0.00s) --- PASS: TestSplitAndCacheMiddleware_StoreAndFetchCacheExtents/fetchCacheExtents()_should_return_a_slice_with_the_same_number_of_input_keys_and_some_extends_filled_up_on_partial_cache_hit (0.00s) --- PASS: TestSplitAndCacheMiddleware_StoreAndFetchCacheExtents/fetchCacheExtents()_should_not_return_an_extent_if_its_key_doesn't_match_the_requested_one_(hash_collision) (0.00s) === RUN TestSplitAndCacheMiddleware_WrapMultipleTimes --- PASS: TestSplitAndCacheMiddleware_WrapMultipleTimes (0.00s) === RUN TestSplitRequests_prepareDownstreamRequests === RUN TestSplitRequests_prepareDownstreamRequests/should_return_an_empty_slice_on_no_downstream_requests === RUN TestSplitRequests_prepareDownstreamRequests/should_inject_ID_and_hints_on_downstream_requests_and_return_them --- PASS: TestSplitRequests_prepareDownstreamRequests (0.00s) --- PASS: TestSplitRequests_prepareDownstreamRequests/should_return_an_empty_slice_on_no_downstream_requests (0.00s) --- PASS: TestSplitRequests_prepareDownstreamRequests/should_inject_ID_and_hints_on_downstream_requests_and_return_them (0.00s) === RUN TestSplitRequests_storeDownstreamResponses === RUN TestSplitRequests_storeDownstreamResponses/should_return_error_if_extra_downstream_responses_are_requested_to_be_stored === RUN TestSplitRequests_storeDownstreamResponses/should_do_nothing_on_no_downstream_requests === RUN TestSplitRequests_storeDownstreamResponses/should_associate_downstream_responses_to_requests === RUN TestSplitRequests_storeDownstreamResponses/should_return_error_if_a_downstream_response_is_missing === RUN TestSplitRequests_storeDownstreamResponses/should_return_error_if_multiple_downstream_responses_have_the_same_ID --- PASS: TestSplitRequests_storeDownstreamResponses (0.00s) --- PASS: TestSplitRequests_storeDownstreamResponses/should_return_error_if_extra_downstream_responses_are_requested_to_be_stored (0.00s) --- PASS: TestSplitRequests_storeDownstreamResponses/should_do_nothing_on_no_downstream_requests (0.00s) --- PASS: TestSplitRequests_storeDownstreamResponses/should_associate_downstream_responses_to_requests (0.00s) --- PASS: TestSplitRequests_storeDownstreamResponses/should_return_error_if_a_downstream_response_is_missing (0.00s) --- PASS: TestSplitRequests_storeDownstreamResponses/should_return_error_if_multiple_downstream_responses_have_the_same_ID (0.00s) === RUN TestNextIntervalBoundary === RUN TestNextIntervalBoundary/0 === RUN TestNextIntervalBoundary/1 === RUN TestNextIntervalBoundary/2 === RUN TestNextIntervalBoundary/3 === RUN TestNextIntervalBoundary/4 === RUN TestNextIntervalBoundary/5 === RUN TestNextIntervalBoundary/6 === RUN TestNextIntervalBoundary/7 === RUN TestNextIntervalBoundary/8 === RUN TestNextIntervalBoundary/9 === RUN TestNextIntervalBoundary/10 === RUN TestNextIntervalBoundary/11 === RUN TestNextIntervalBoundary/12 === RUN TestNextIntervalBoundary/13 === RUN TestNextIntervalBoundary/14 === RUN TestNextIntervalBoundary/15 --- PASS: TestNextIntervalBoundary (0.01s) --- PASS: TestNextIntervalBoundary/0 (0.00s) --- PASS: TestNextIntervalBoundary/1 (0.00s) --- PASS: TestNextIntervalBoundary/2 (0.00s) --- PASS: TestNextIntervalBoundary/3 (0.00s) --- PASS: TestNextIntervalBoundary/4 (0.00s) --- PASS: TestNextIntervalBoundary/5 (0.00s) --- PASS: TestNextIntervalBoundary/6 (0.00s) --- PASS: TestNextIntervalBoundary/7 (0.00s) --- PASS: TestNextIntervalBoundary/8 (0.00s) --- PASS: TestNextIntervalBoundary/9 (0.00s) --- PASS: TestNextIntervalBoundary/10 (0.00s) --- PASS: TestNextIntervalBoundary/11 (0.00s) --- PASS: TestNextIntervalBoundary/12 (0.00s) --- PASS: TestNextIntervalBoundary/13 (0.00s) --- PASS: TestNextIntervalBoundary/14 (0.00s) --- PASS: TestNextIntervalBoundary/15 (0.00s) === RUN TestSplitQueryByInterval === RUN TestSplitQueryByInterval/0:_start:_0,_end:_3600000,_step:_15000 === RUN TestSplitQueryByInterval/1:_start:_0,_end:_3600000,_step:_15000 === RUN TestSplitQueryByInterval/2:_start:_0,_end:_86400000,_step:_15000 === RUN TestSplitQueryByInterval/3:_start:_0,_end:_10800000,_step:_15000 === RUN TestSplitQueryByInterval/4:_start:_0,_end:_172800000,_step:_15000 === RUN TestSplitQueryByInterval/5:_start:_0,_end:_21600000,_step:_15000 === RUN TestSplitQueryByInterval/6:_start:_10800000,_end:_259200000,_step:_15000 === RUN TestSplitQueryByInterval/7:_start:_7200000,_end:_32400000,_step:_15000 === RUN TestSplitQueryByInterval/8:_start:_1634255280000,_end:_1634256180000,_step:_300000 === RUN TestSplitQueryByInterval/9:_start:_1634255280000,_end:_1634256000000,_step:_360000 === RUN TestSplitQueryByInterval/10:_start:_1634248800000,_end:_1634508000000,_step:_86400000 === RUN TestSplitQueryByInterval/11:_start:_1634256000000,_end:_1634515200000,_step:_86400000 === RUN TestSplitQueryByInterval/12:_start:_1634335200000,_end:_1634875200000,_step:_108000000 === RUN TestSplitQueryByInterval/13:_start:_1634277600000,_end:_1634479200000,_step:_43200000 === RUN TestSplitQueryByInterval/14:_start:_1634277600000,_end:_1634493600000,_step:_43200000 === RUN TestSplitQueryByInterval/15:_start:_1634277600000,_end:_1634493600000,_step:_36000000 === RUN TestSplitQueryByInterval/16:_start:_1634277600000,_end:_1634457600000,_step:_36000000 --- PASS: TestSplitQueryByInterval (0.01s) --- PASS: TestSplitQueryByInterval/0:_start:_0,_end:_3600000,_step:_15000 (0.00s) --- PASS: TestSplitQueryByInterval/1:_start:_0,_end:_3600000,_step:_15000 (0.00s) --- PASS: TestSplitQueryByInterval/2:_start:_0,_end:_86400000,_step:_15000 (0.00s) --- PASS: TestSplitQueryByInterval/3:_start:_0,_end:_10800000,_step:_15000 (0.00s) --- PASS: TestSplitQueryByInterval/4:_start:_0,_end:_172800000,_step:_15000 (0.00s) --- PASS: TestSplitQueryByInterval/5:_start:_0,_end:_21600000,_step:_15000 (0.01s) --- PASS: TestSplitQueryByInterval/6:_start:_10800000,_end:_259200000,_step:_15000 (0.00s) --- PASS: TestSplitQueryByInterval/7:_start:_7200000,_end:_32400000,_step:_15000 (0.00s) --- PASS: TestSplitQueryByInterval/8:_start:_1634255280000,_end:_1634256180000,_step:_300000 (0.00s) --- PASS: TestSplitQueryByInterval/9:_start:_1634255280000,_end:_1634256000000,_step:_360000 (0.00s) --- PASS: TestSplitQueryByInterval/10:_start:_1634248800000,_end:_1634508000000,_step:_86400000 (0.00s) --- PASS: TestSplitQueryByInterval/11:_start:_1634256000000,_end:_1634515200000,_step:_86400000 (0.00s) --- PASS: TestSplitQueryByInterval/12:_start:_1634335200000,_end:_1634875200000,_step:_108000000 (0.00s) --- PASS: TestSplitQueryByInterval/13:_start:_1634277600000,_end:_1634479200000,_step:_43200000 (0.00s) --- PASS: TestSplitQueryByInterval/14:_start:_1634277600000,_end:_1634493600000,_step:_43200000 (0.00s) --- PASS: TestSplitQueryByInterval/15:_start:_1634277600000,_end:_1634493600000,_step:_36000000 (0.00s) --- PASS: TestSplitQueryByInterval/16:_start:_1634277600000,_end:_1634457600000,_step:_36000000 (0.00s) === RUN Test_evaluateAtModifier === RUN Test_evaluateAtModifier/topk(5,_rate(http_requests_total[1h]_@_start())) === PAUSE Test_evaluateAtModifier/topk(5,_rate(http_requests_total[1h]_@_start())) === RUN Test_evaluateAtModifier/topk(5,_rate(http_requests_total[1h]_@_0)) === PAUSE Test_evaluateAtModifier/topk(5,_rate(http_requests_total[1h]_@_0)) === RUN Test_evaluateAtModifier/http_requests_total[1h]_@_10.001 === PAUSE Test_evaluateAtModifier/http_requests_total[1h]_@_10.001 === RUN Test_evaluateAtModifier/min_over_time(_____sum_by(cluster)_(______rate(http_requests_total[5m]_@_end())_____)[10m:]____)____or____max_over_time(_____stddev_over_time(______deriv(_______rate(http_requests_total[10m]_@_start())______[5m:1m])_____[2m:])____[10m:]) === PAUSE Test_evaluateAtModifier/min_over_time(_____sum_by(cluster)_(______rate(http_requests_total[5m]_@_end())_____)[10m:]____)____or____max_over_time(_____stddev_over_time(______deriv(_______rate(http_requests_total[10m]_@_start())______[5m:1m])_____[2m:])____[10m:]) === RUN Test_evaluateAtModifier/sum_by_(foo)_(bar[buzz]) === PAUSE Test_evaluateAtModifier/sum_by_(foo)_(bar[buzz]) === CONT Test_evaluateAtModifier/topk(5,_rate(http_requests_total[1h]_@_start())) === CONT Test_evaluateAtModifier/sum_by_(foo)_(bar[buzz]) === CONT Test_evaluateAtModifier/http_requests_total[1h]_@_10.001 === CONT Test_evaluateAtModifier/topk(5,_rate(http_requests_total[1h]_@_0)) === CONT Test_evaluateAtModifier/min_over_time(_____sum_by(cluster)_(______rate(http_requests_total[5m]_@_end())_____)[10m:]____)____or____max_over_time(_____stddev_over_time(______deriv(_______rate(http_requests_total[10m]_@_start())______[5m:1m])_____[2m:])____[10m:]) --- PASS: Test_evaluateAtModifier (0.00s) --- PASS: Test_evaluateAtModifier/topk(5,_rate(http_requests_total[1h]_@_start())) (0.00s) --- PASS: Test_evaluateAtModifier/sum_by_(foo)_(bar[buzz]) (0.00s) --- PASS: Test_evaluateAtModifier/http_requests_total[1h]_@_10.001 (0.00s) --- PASS: Test_evaluateAtModifier/topk(5,_rate(http_requests_total[1h]_@_0)) (0.00s) --- PASS: Test_evaluateAtModifier/min_over_time(_____sum_by(cluster)_(______rate(http_requests_total[5m]_@_end())_____)[10m:]____)____or____max_over_time(_____stddev_over_time(______deriv(_______rate(http_requests_total[10m]_@_start())______[5m:1m])_____[2m:])____[10m:]) (0.00s) === RUN TestSplitAndCacheMiddlewareLowerTTL --- PASS: TestSplitAndCacheMiddlewareLowerTTL (0.00s) === RUN TestInstantQuerySplittingCorrectness === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sum_over_time === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sum_over_time === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/ceil(sum(sum_over_time())) === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/ceil(sum(sum_over_time())) === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/histogram_quantile === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/histogram_quantile === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/min(rate) === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/min(rate) === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sum(sum_over_time_@_end()) === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sum(sum_over_time_@_end()) === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/time() === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/time() === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/exp === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/exp === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/subquery_sum_grouping_'by' === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/subquery_sum_grouping_'by' === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/absent_over_time === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/absent_over_time === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/min_over_time === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/min_over_time === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/clamp === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/clamp === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/label_replace === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/label_replace === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/stddev(rate) === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/stddev(rate) === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sum(avg_over_time) === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sum(avg_over_time) === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sum_over_time[3m]_offset_30s === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sum_over_time[3m]_offset_30s === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/avg(rate)_grouping_'by' === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/avg(rate)_grouping_'by' === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/count_over_time[3m]_offset_-30s === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/count_over_time[3m]_offset_-30s === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/subquery_sum(rate) === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/subquery_sum(rate) === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/changes === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/changes === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/label_join === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/label_join === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sort === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sort === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/max(rate)_grouping_'by' === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/max(rate)_grouping_'by' === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sum(sum_over_time_+_count_over_time) === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sum(sum_over_time_+_count_over_time) === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sum_over_time[3m]_offset_3m === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sum_over_time[3m]_offset_3m === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/vector(10) === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/vector(10) === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/log2 === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/log2 === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sqrt === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sqrt === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/avg(rate) === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/avg(rate) === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/max(rate) === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/max(rate) === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/subquery_sum_over_time === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/subquery_sum_over_time === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sum(rate)_grouping_'by' === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sum(rate)_grouping_'by' === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/count_values(count_over_time) === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/count_values(count_over_time) === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/rate(3m)_/_rate(3m)_>_0.5 === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/rate(3m)_/_rate(3m)_>_0.5 === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/histogram_quantile()_grouping_'by' === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/histogram_quantile()_grouping_'by' === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/ceil(sum(sum_over_time())_+_sum(sum_over_time()))_and_only_right_leg_of_the_binary_operation_is_splittable === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/ceil(sum(sum_over_time())_+_sum(sum_over_time()))_and_only_right_leg_of_the_binary_operation_is_splittable === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/count(rate) === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/count(rate) === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/avg_over_time[3m]_offset_-1m === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/avg_over_time[3m]_offset_-1m === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sum_over_time_@_start() === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sum_over_time_@_start() === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/present_over_time === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/present_over_time === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/rate_/_rate === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/rate_/_rate === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/delta === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/delta === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/stdvar_over_time === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/stdvar_over_time === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/rate === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/rate === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/ceil(sum(sum_over_time())_+_sum(sum_over_time()))_and_only_left_leg_of_the_binary_operation_is_splittable === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/ceil(sum(sum_over_time())_+_sum(sum_over_time()))_and_only_left_leg_of_the_binary_operation_is_splittable === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/topk(rate) === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/topk(rate) === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sum(max(rate)) === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sum(max(rate)) === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/deriv === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/deriv === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/resets === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/resets === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/stddev_over_time === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/stddev_over_time === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/avg(avg_over_time_@_`start`) === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/avg(avg_over_time_@_`start`) === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/floor === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/floor === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/min(rate)_grouping_'by' === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/min(rate)_grouping_'by' === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sum(rate) === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sum(rate) === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/histogram_quantile()_grouping_only_'by'_le === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/histogram_quantile()_grouping_only_'by'_le === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/increase === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/increase === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sgn === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sgn === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/topk(sum(rate)) === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/topk(sum(rate)) === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/(avg_over_time) === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/(avg_over_time) === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/count_over_time[3m]_offset_-2m === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/count_over_time[3m]_offset_-2m === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/histogram_quantile()_grouping_'without' === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/histogram_quantile()_grouping_'without' === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/idelta === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/idelta === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/last_over_time === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/last_over_time === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/predict_linear === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/predict_linear === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/count_over_time === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/count_over_time === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/clamp_min === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/clamp_min === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/ln === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/ln === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sum(rate()_or_rate()) === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sum(rate()_or_rate()) === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/rate_/_10 === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/rate_/_10 === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/holt_winters === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/holt_winters === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/quantile_over_time === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/quantile_over_time === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/ceil(sum(sum_over_time())_+_sum(sum_over_time()))_and_both_legs_of_the_binary_operation_are_splittable === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/ceil(sum(sum_over_time())_+_sum(sum_over_time()))_and_both_legs_of_the_binary_operation_are_splittable === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sort_desc === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sort_desc === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/10_/_rate === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/10_/_rate === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/rate(1m)_/_rate(subquery)_>_0.5 === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/rate(1m)_/_rate(subquery)_>_0.5 === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/avg_over_time === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/avg_over_time === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/absent === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/absent === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/avg_over_time[3m]_offset_5m === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/avg_over_time[3m]_offset_5m === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/max_over_time_@_`start`_offset_1m) === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/max_over_time_@_`start`_offset_1m) === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/max_over_time === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/max_over_time === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/clamp_max === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/clamp_max === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/round === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/round === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/count(rate)_grouping_'by' === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/count(rate)_grouping_'by' === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/irate === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/irate === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/min_over_time_offset_1m_@_`start`) === PAUSE TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/min_over_time_offset_1m_@_`start`) === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sum_over_time === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sum_over_time/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/rate === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/stdvar_over_time === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/stdvar_over_time/*querymiddleware.PrometheusInstantQueryRequest === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/rate/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sum(sum_over_time_+_count_over_time) === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sum(sum_over_time_+_count_over_time)/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/label_replace === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/label_replace/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/max(rate)_grouping_'by' === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/max(rate)_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sort === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sort/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/label_join === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/label_join/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/changes === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/changes/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/subquery_sum(rate) === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/subquery_sum(rate)/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/count_over_time[3m]_offset_-30s === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/count_over_time[3m]_offset_-30s/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/avg(rate)_grouping_'by' === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/avg(rate)_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sum_over_time[3m]_offset_30s === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sum_over_time[3m]_offset_30s/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sum(avg_over_time) === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sum(avg_over_time)/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/stddev(rate) === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/stddev(rate)/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/exp === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/exp/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/clamp === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/clamp/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/min_over_time === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/min_over_time/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/absent_over_time === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/absent_over_time/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/subquery_sum_grouping_'by' === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/subquery_sum_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/min(rate) === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/min(rate)/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/time() === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/time()/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sum(sum_over_time_@_end()) === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sum(sum_over_time_@_end())/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/histogram_quantile === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/histogram_quantile/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/count_over_time === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/count_over_time/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/min_over_time_offset_1m_@_`start`) === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/min_over_time_offset_1m_@_`start`)/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/irate === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/irate/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/count(rate)_grouping_'by' === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/count(rate)_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/round === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/round/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/clamp_max === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/clamp_max/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/max_over_time === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/max_over_time/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/max_over_time_@_`start`_offset_1m) === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/max_over_time_@_`start`_offset_1m)/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/avg_over_time[3m]_offset_5m === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/avg_over_time[3m]_offset_5m/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/absent === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/absent/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/avg_over_time === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/avg_over_time/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/rate(1m)_/_rate(subquery)_>_0.5 === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/rate(1m)_/_rate(subquery)_>_0.5/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/10_/_rate === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/10_/_rate/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sort_desc === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/sort_desc/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/ceil(sum(sum_over_time())_+_sum(sum_over_time()))_and_both_legs_of_the_binary_operation_are_splittable === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/ceil(sum(sum_over_time())_+_sum(sum_over_time()))_and_both_legs_of_the_binary_operation_are_splittable/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/quantile_over_time === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/quantile_over_time/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/holt_winters === RUN TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/holt_winters/*querymiddleware.PrometheusInstantQueryRequest === CONT TestInstantQuerySplittingCorrectness/start=2020-01-01T03:00:00.100Z/rate_/_10 === RUN T