==> Building on turtonator ==> Attempt to run pkgver()... ==> Checking for remote environment... ==> Syncing package to remote host... sending incremental file list created directory packages/mimir ./ PKGBUILD 2,430 100% 0.00kB/s 0:00:00 2,430 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=5/7) config.yaml 38 100% 37.11kB/s 0:00:00 38 100% 37.11kB/s 0:00:00 (xfr#2, to-chk=4/7) mimir-2.8.0-1.log 528 100% 515.62kB/s 0:00:00 528 100% 515.62kB/s 0:00:00 (xfr#3, to-chk=3/7) systemd.service 1,011 100% 987.30kB/s 0:00:00 1,011 100% 987.30kB/s 0:00:00 (xfr#4, to-chk=2/7) sysusers.conf 45 100% 43.95kB/s 0:00:00 45 100% 43.95kB/s 0:00:00 (xfr#5, to-chk=1/7) tmpfiles.conf 34 100% 33.20kB/s 0:00:00 34 100% 33.20kB/s 0:00:00 (xfr#6, to-chk=0/7) sent 2,597 bytes received 170 bytes 5,534.00 bytes/sec total size is 3,807 speedup is 1.38 ==> Patching arch to riscv64... ==> Running extra-riscv64-build -- -d /home/felix/packages/riscv64-pkg-cache:/var/cache/pacman/pkg -l root1 on remote host... [?25l:: Synchronizing package databases... core downloading... extra downloading... community downloading... :: Starting full system upgrade... resolving dependencies... looking for conflicting packages... Package (1) Old Version New Version Net Change Download Size core/glibc 2.37-2.1 2.37-3 0.01 MiB 11.04 MiB Total Download Size: 11.04 MiB Total Installed Size: 55.73 MiB Net Upgrade Size: 0.01 MiB :: Proceed with installation? [Y/n] :: Retrieving packages... glibc-2.37-3-riscv64 downloading... checking keyring... checking package integrity... loading package files... checking for file conflicts... :: Processing package changes... upgrading glibc... Generating locales... en_US.UTF-8... done de_DE.UTF-8... done Generation complete. :: Running post-transaction hooks... (1/1) Updating the info directory file... [?25h==> Building in chroot for [extra] (riscv64)... ==> Locking clean chroot [/var/lib/archbuild/extra-riscv64/root]...done ==> Synchronizing chroot copy [/var/lib/archbuild/extra-riscv64/root] -> [root1]...done ==> Making package: mimir 2.8.0-1 (Thu May 11 18:16:33 2023) ==> Retrieving sources...  -> Cloning mimir git repo... Cloning into bare repository '/home/felix/packages/mimir/mimir'... remote: Enumerating objects: 251390, done. remote: Counting objects: 0% (1/11616) remote: Counting objects: 1% (117/11616) remote: Counting objects: 2% (233/11616) remote: Counting objects: 3% (349/11616) remote: Counting objects: 4% (465/11616) remote: Counting objects: 5% (581/11616) remote: Counting objects: 6% (697/11616) remote: Counting objects: 7% (814/11616) remote: Counting objects: 8% (930/11616) remote: Counting objects: 9% (1046/11616) remote: Counting objects: 10% (1162/11616) remote: Counting objects: 11% (1278/11616) remote: Counting objects: 12% (1394/11616) remote: Counting objects: 13% (1511/11616) remote: Counting objects: 14% (1627/11616) remote: Counting objects: 15% (1743/11616) remote: Counting objects: 16% (1859/11616) remote: Counting objects: 17% (1975/11616) remote: Counting objects: 18% (2091/11616) remote: Counting objects: 19% (2208/11616) remote: Counting objects: 20% (2324/11616) remote: Counting objects: 21% (2440/11616) remote: Counting objects: 22% (2556/11616) remote: Counting objects: 23% (2672/11616) remote: Counting objects: 24% (2788/11616) remote: Counting objects: 25% (2904/11616) remote: Counting objects: 26% (3021/11616) remote: Counting objects: 27% (3137/11616) remote: Counting objects: 28% (3253/11616) remote: Counting objects: 29% (3369/11616) remote: Counting objects: 30% (3485/11616) remote: Counting objects: 31% (3601/11616) remote: Counting objects: 32% (3718/11616) remote: Counting objects: 33% (3834/11616) remote: Counting objects: 34% (3950/11616) remote: Counting objects: 35% (4066/11616) remote: Counting objects: 36% (4182/11616) remote: Counting objects: 37% (4298/11616) remote: Counting objects: 38% (4415/11616) remote: Counting objects: 39% (4531/11616) remote: Counting objects: 40% (4647/11616) remote: Counting objects: 41% (4763/11616) remote: Counting objects: 42% (4879/11616) remote: Counting objects: 43% (4995/11616) remote: Counting objects: 44% (5112/11616) remote: Counting objects: 45% (5228/11616) remote: Counting objects: 46% (5344/11616) remote: Counting objects: 47% (5460/11616) remote: Counting objects: 48% (5576/11616) remote: Counting objects: 49% (5692/11616) remote: Counting objects: 50% (5808/11616) remote: Counting objects: 51% (5925/11616) remote: Counting objects: 52% (6041/11616) remote: Counting objects: 53% (6157/11616) remote: Counting objects: 54% (6273/11616) remote: Counting objects: 55% (6389/11616) remote: Counting objects: 56% (6505/11616) remote: Counting objects: 57% (6622/11616) remote: Counting objects: 58% (6738/11616) remote: Counting objects: 59% (6854/11616) remote: Counting objects: 60% (6970/11616) remote: Counting objects: 61% (7086/11616) remote: Counting objects: 62% (7202/11616) remote: Counting objects: 63% (7319/11616) remote: Counting objects: 64% (7435/11616) remote: Counting objects: 65% (7551/11616) remote: Counting objects: 66% (7667/11616) remote: Counting objects: 67% (7783/11616) remote: Counting objects: 68% (7899/11616) remote: Counting objects: 69% (8016/11616) remote: Counting objects: 70% (8132/11616) remote: Counting objects: 71% (8248/11616) remote: Counting objects: 72% (8364/11616) remote: Counting objects: 73% (8480/11616) remote: Counting objects: 74% (8596/11616) remote: Counting objects: 75% (8712/11616) remote: Counting objects: 76% (8829/11616) remote: Counting objects: 77% (8945/11616) remote: Counting objects: 78% (9061/11616) remote: Counting objects: 79% (9177/11616) remote: Counting objects: 80% (9293/11616) remote: Counting objects: 81% (9409/11616) remote: Counting objects: 82% (9526/11616) remote: Counting objects: 83% (9642/11616) remote: Counting objects: 84% (9758/11616) remote: Counting objects: 85% (9874/11616) remote: Counting objects: 86% (9990/11616) remote: Counting objects: 87% (10106/11616) remote: Counting objects: 88% (10223/11616) remote: Counting objects: 89% (10339/11616) remote: Counting objects: 90% (10455/11616) remote: Counting objects: 91% (10571/11616) remote: Counting objects: 92% (10687/11616) remote: Counting objects: 93% (10803/11616) remote: Counting objects: 94% (10920/11616) remote: Counting objects: 95% (11036/11616) remote: Counting objects: 96% (11152/11616) remote: Counting objects: 97% (11268/11616) remote: Counting objects: 98% (11384/11616) remote: Counting objects: 99% (11500/11616) remote: Counting objects: 100% (11616/11616) remote: Counting objects: 100% (11616/11616), done. remote: Compressing objects: 0% (1/4403) remote: Compressing objects: 1% (45/4403) remote: Compressing objects: 2% (89/4403) remote: Compressing objects: 3% (133/4403) remote: Compressing objects: 4% (177/4403) remote: Compressing objects: 5% (221/4403) remote: Compressing objects: 6% (265/4403) remote: Compressing objects: 7% (309/4403) remote: Compressing objects: 7% (351/4403) remote: Compressing objects: 8% (353/4403) remote: Compressing objects: 9% (397/4403) remote: Compressing objects: 10% (441/4403) remote: Compressing objects: 11% (485/4403) remote: Compressing objects: 12% (529/4403) remote: Compressing objects: 13% (573/4403) remote: Compressing objects: 14% (617/4403) remote: Compressing objects: 15% (661/4403) remote: Compressing objects: 16% (705/4403) remote: Compressing objects: 17% (749/4403) remote: Compressing objects: 18% (793/4403) remote: Compressing objects: 19% (837/4403) remote: Compressing objects: 20% (881/4403) remote: Compressing objects: 21% (925/4403) remote: Compressing objects: 22% (969/4403) remote: Compressing objects: 23% (1013/4403) remote: Compressing objects: 24% (1057/4403) remote: Compressing objects: 25% (1101/4403) remote: Compressing objects: 26% (1145/4403) remote: Compressing objects: 27% (1189/4403) remote: Compressing objects: 28% (1233/4403) remote: Compressing objects: 29% (1277/4403) remote: Compressing objects: 30% (1321/4403) remote: Compressing objects: 31% (1365/4403) remote: Compressing objects: 32% (1409/4403) remote: Compressing objects: 33% (1453/4403) remote: Compressing objects: 34% (1498/4403) remote: Compressing objects: 35% (1542/4403) remote: Compressing objects: 36% (1586/4403) remote: Compressing objects: 37% (1630/4403) remote: Compressing objects: 38% (1674/4403) remote: Compressing objects: 39% (1718/4403) remote: Compressing objects: 40% (1762/4403) remote: Compressing objects: 41% (1806/4403) remote: Compressing objects: 42% (1850/4403) remote: Compressing objects: 43% (1894/4403) remote: Compressing objects: 44% (1938/4403) remote: Compressing objects: 45% (1982/4403) remote: Compressing objects: 46% (2026/4403) remote: Compressing objects: 47% (2070/4403) remote: Compressing objects: 48% (2114/4403) remote: Compressing objects: 49% (2158/4403) remote: Compressing objects: 50% (2202/4403) remote: Compressing objects: 51% (2246/4403) remote: Compressing objects: 52% (2290/4403) remote: Compressing objects: 53% (2334/4403) remote: Compressing objects: 54% (2378/4403) remote: Compressing objects: 55% (2422/4403) remote: Compressing objects: 56% (2466/4403) remote: Compressing objects: 57% (2510/4403) remote: Compressing objects: 58% (2554/4403) remote: Compressing objects: 59% (2598/4403) remote: Compressing objects: 60% (2642/4403) remote: Compressing objects: 61% (2686/4403) remote: Compressing objects: 62% (2730/4403) remote: Compressing objects: 63% (2774/4403) remote: Compressing objects: 64% (2818/4403) remote: Compressing objects: 65% (2862/4403) remote: Compressing objects: 66% (2906/4403) remote: Compressing objects: 67% (2951/4403) remote: Compressing objects: 68% (2995/4403) remote: Compressing objects: 69% (3039/4403) remote: Compressing objects: 70% (3083/4403) remote: Compressing objects: 71% (3127/4403) remote: Compressing objects: 72% (3171/4403) remote: Compressing objects: 73% (3215/4403) remote: Compressing objects: 74% (3259/4403) remote: Compressing objects: 75% (3303/4403) remote: Compressing objects: 76% (3347/4403) remote: Compressing objects: 77% (3391/4403) remote: Compressing objects: 78% (3435/4403) remote: Compressing objects: 79% (3479/4403) remote: Compressing objects: 80% (3523/4403) remote: Compressing objects: 81% (3567/4403) remote: Compressing objects: 82% (3611/4403) remote: Compressing objects: 83% (3655/4403) remote: Compressing objects: 84% (3699/4403) remote: Compressing objects: 85% (3743/4403) remote: Compressing objects: 86% (3787/4403) remote: Compressing objects: 87% (3831/4403) remote: Compressing objects: 88% (3875/4403) remote: Compressing objects: 89% (3919/4403) remote: Compressing objects: 90% (3963/4403) remote: Compressing objects: 91% (4007/4403) remote: Compressing objects: 92% (4051/4403) remote: Compressing objects: 93% (4095/4403) remote: Compressing objects: 94% (4139/4403) remote: Compressing objects: 95% (4183/4403) remote: Compressing objects: 96% (4227/4403) remote: Compressing objects: 97% (4271/4403) remote: Compressing objects: 98% (4315/4403) remote: Compressing objects: 99% (4359/4403) remote: Compressing objects: 100% (4403/4403) remote: Compressing objects: 100% (4403/4403), done. Receiving objects: 0% (1/251390) Receiving objects: 1% (2514/251390), 1.23 MiB | 2.45 MiB/s Receiving objects: 1% (3013/251390), 1.23 MiB | 2.45 MiB/s Receiving objects: 2% (5028/251390), 2.16 MiB | 2.15 MiB/s Receiving objects: 2% (6806/251390), 3.33 MiB | 2.21 MiB/s Receiving objects: 3% (7542/251390), 3.33 MiB | 2.21 MiB/s Receiving objects: 4% (10056/251390), 4.23 MiB | 2.11 MiB/s Receiving objects: 4% (11097/251390), 5.34 MiB | 2.13 MiB/s Receiving objects: 4% (11653/251390), 7.93 MiB | 2.26 MiB/s Receiving objects: 5% (12570/251390), 9.35 MiB | 2.33 MiB/s Receiving objects: 5% (14431/251390), 10.60 MiB | 2.35 MiB/s Receiving objects: 5% (14437/251390), 11.09 MiB | 808.00 KiB/s Receiving objects: 6% (15084/251390), 12.20 MiB | 822.00 KiB/s Receiving objects: 6% (15570/251390), 12.20 MiB | 822.00 KiB/s Receiving objects: 6% (16364/251390), 14.18 MiB | 815.00 KiB/s Receiving objects: 7% (17598/251390), 15.32 MiB | 817.00 KiB/s Receiving objects: 7% (18246/251390), 16.40 MiB | 817.00 KiB/s Receiving objects: 8% (20112/251390), 19.02 MiB | 792.00 KiB/s Receiving objects: 8% (20408/251390), 19.02 MiB | 792.00 KiB/s Receiving objects: 8% (21854/251390), 21.47 MiB | 2.30 MiB/s Receiving objects: 9% (22626/251390), 22.66 MiB | 2.32 MiB/s Receiving objects: 9% (23663/251390), 23.90 MiB | 2.34 MiB/s Receiving objects: 10% (25139/251390), 24.94 MiB | 2.38 MiB/s Receiving objects: 10% (25708/251390), 26.27 MiB | 2.43 MiB/s Receiving objects: 10% (27094/251390), 28.55 MiB | 2.46 MiB/s Receiving objects: 11% (27653/251390), 29.71 MiB | 2.37 MiB/s Receiving objects: 11% (29437/251390), 30.84 MiB | 2.34 MiB/s Receiving objects: 12% (30167/251390), 33.71 MiB | 2.45 MiB/s Receiving objects: 12% (31530/251390), 33.71 MiB | 2.45 MiB/s Receiving objects: 13% (32681/251390), 35.35 MiB | 2.54 MiB/s Receiving objects: 13% (33745/251390), 36.98 MiB | 2.67 MiB/s Receiving objects: 14% (35195/251390), 38.33 MiB | 2.67 MiB/s Receiving objects: 14% (37189/251390), 39.70 MiB | 2.76 MiB/s Receiving objects: 15% (37709/251390), 41.14 MiB | 2.79 MiB/s Receiving objects: 15% (38678/251390), 42.49 MiB | 2.83 MiB/s Receiving objects: 16% (40223/251390), 43.99 MiB | 2.91 MiB/s Receiving objects: 16% (40614/251390), 45.40 MiB | 2.93 MiB/s Receiving objects: 16% (42581/251390), 47.99 MiB | 2.80 MiB/s Receiving objects: 17% (42737/251390), 47.99 MiB | 2.80 MiB/s Receiving objects: 17% (44400/251390), 51.13 MiB | 2.84 MiB/s Receiving objects: 18% (45251/251390), 51.13 MiB | 2.84 MiB/s Receiving objects: 18% (47157/251390), 54.11 MiB | 2.88 MiB/s Receiving objects: 19% (47765/251390), 54.11 MiB | 2.88 MiB/s Receiving objects: 20% (50278/251390), 56.62 MiB | 2.80 MiB/s Receiving objects: 20% (51331/251390), 56.62 MiB | 2.80 MiB/s Receiving objects: 20% (52559/251390), 58.82 MiB | 2.71 MiB/s Receiving objects: 21% (52792/251390), 58.82 MiB | 2.71 MiB/s Receiving objects: 21% (54009/251390), 62.68 MiB | 2.56 MiB/s Receiving objects: 21% (54009/251390), 62.94 MiB | 734.00 KiB/s Receiving objects: 21% (54009/251390), 63.96 MiB | 689.00 KiB/s Receiving objects: 21% (54009/251390), 66.18 MiB | 591.00 KiB/s Receiving objects: 21% (54009/251390), 68.70 MiB | 611.00 KiB/s Receiving objects: 21% (54009/251390), 71.49 MiB | 619.00 KiB/s Receiving objects: 22% (55306/251390), 72.76 MiB | 623.00 KiB/s Receiving objects: 22% (56110/251390), 72.76 MiB | 623.00 KiB/s Receiving objects: 23% (57820/251390), 72.76 MiB | 623.00 KiB/s Receiving objects: 23% (59420/251390), 75.83 MiB | 1.85 MiB/s Receiving objects: 24% (60334/251390), 75.83 MiB | 1.85 MiB/s Receiving objects: 24% (60841/251390), 78.98 MiB | 2.50 MiB/s Receiving objects: 25% (62848/251390), 78.98 MiB | 2.50 MiB/s Receiving objects: 25% (65196/251390), 80.69 MiB | 2.66 MiB/s Receiving objects: 26% (65362/251390), 80.69 MiB | 2.66 MiB/s Receiving objects: 27% (67876/251390), 81.94 MiB | 2.64 MiB/s Receiving objects: 28% (70390/251390), 83.85 MiB | 2.74 MiB/s Receiving objects: 28% (71770/251390), 83.85 MiB | 2.74 MiB/s Receiving objects: 29% (72904/251390), 83.85 MiB | 2.74 MiB/s Receiving objects: 30% (75417/251390), 87.02 MiB | 2.86 MiB/s Receiving objects: 30% (77482/251390), 87.02 MiB | 2.86 MiB/s Receiving objects: 31% (77931/251390), 90.07 MiB | 2.83 MiB/s Receiving objects: 31% (78813/251390), 90.07 MiB | 2.83 MiB/s Receiving objects: 32% (80445/251390), 90.07 MiB | 2.83 MiB/s Receiving objects: 32% (81959/251390), 92.40 MiB | 2.96 MiB/s Receiving objects: 33% (82959/251390), 92.40 MiB | 2.96 MiB/s Receiving objects: 34% (85473/251390), 93.38 MiB | 2.80 MiB/s Receiving objects: 35% (87987/251390), 93.38 MiB | 2.80 MiB/s Receiving objects: 35% (90010/251390), 94.87 MiB | 2.86 MiB/s Receiving objects: 36% (90501/251390), 94.87 MiB | 2.86 MiB/s Receiving objects: 37% (93015/251390), 96.06 MiB | 2.69 MiB/s Receiving objects: 37% (94430/251390), 97.23 MiB | 2.67 MiB/s Receiving objects: 38% (95529/251390), 97.23 MiB | 2.67 MiB/s Receiving objects: 39% (98043/251390), 98.40 MiB | 2.52 MiB/s Receiving objects: 39% (98927/251390), 100.05 MiB | 2.54 MiB/s Receiving objects: 40% (100556/251390), 101.75 MiB | 2.58 MiB/s Receiving objects: 41% (103070/251390), 101.75 MiB | 2.58 MiB/s Receiving objects: 41% (104400/251390), 104.00 MiB | 2.83 MiB/s Receiving objects: 42% (105584/251390), 104.00 MiB | 2.83 MiB/s Receiving objects: 43% (108098/251390), 105.80 MiB | 2.96 MiB/s Receiving objects: 43% (108991/251390), 107.64 MiB | 3.15 MiB/s Receiving objects: 44% (110612/251390), 109.09 MiB | 3.15 MiB/s Receiving objects: 44% (111196/251390), 112.64 MiB | 3.42 MiB/s Receiving objects: 44% (111196/251390), 115.50 MiB | 3.43 MiB/s Receiving objects: 44% (111196/251390), 118.25 MiB | 3.16 MiB/s Receiving objects: 44% (111196/251390), 121.05 MiB | 2.97 MiB/s Receiving objects: 44% (111196/251390), 123.62 MiB | 2.75 MiB/s Receiving objects: 44% (111196/251390), 125.79 MiB | 2.55 MiB/s Receiving objects: 44% (111196/251390), 128.07 MiB | 2.44 MiB/s Receiving objects: 45% (113126/251390), 128.07 MiB | 2.44 MiB/s Receiving objects: 46% (115640/251390), 128.07 MiB | 2.44 MiB/s Receiving objects: 46% (116588/251390), 129.25 MiB | 2.43 MiB/s Receiving objects: 47% (118154/251390), 129.25 MiB | 2.43 MiB/s Receiving objects: 48% (120668/251390), 130.43 MiB | 2.37 MiB/s Receiving objects: 48% (122655/251390), 132.02 MiB | 2.43 MiB/s Receiving objects: 49% (123182/251390), 132.02 MiB | 2.43 MiB/s Receiving objects: 50% (125695/251390), 133.19 MiB | 2.42 MiB/s Receiving objects: 50% (126566/251390), 134.39 MiB | 2.39 MiB/s Receiving objects: 51% (128209/251390), 134.39 MiB | 2.39 MiB/s Receiving objects: 52% (130723/251390), 135.77 MiB | 2.41 MiB/s Receiving objects: 52% (132014/251390), 137.29 MiB | 2.54 MiB/s Receiving objects: 53% (133237/251390), 137.29 MiB | 2.54 MiB/s Receiving objects: 53% (134283/251390), 140.14 MiB | 2.66 MiB/s Receiving objects: 54% (135751/251390), 140.14 MiB | 2.66 MiB/s Receiving objects: 55% (138265/251390), 142.77 MiB | 2.71 MiB/s Receiving objects: 55% (138271/251390), 142.77 MiB | 2.71 MiB/s Receiving objects: 56% (140779/251390), 142.77 MiB | 2.71 MiB/s Receiving objects: 57% (143293/251390), 144.04 MiB | 2.64 MiB/s Receiving objects: 57% (144891/251390), 145.33 MiB | 2.66 MiB/s Receiving objects: 58% (145807/251390), 145.33 MiB | 2.66 MiB/s Receiving objects: 58% (148265/251390), 147.52 MiB | 2.59 MiB/s Receiving objects: 59% (148321/251390), 147.52 MiB | 2.59 MiB/s Receiving objects: 60% (150834/251390), 148.50 MiB | 2.47 MiB/s Receiving objects: 60% (152212/251390), 149.82 MiB | 738.00 KiB/s Receiving objects: 61% (153348/251390), 149.82 MiB | 738.00 KiB/s Receiving objects: 62% (155862/251390), 149.82 MiB | 738.00 KiB/s Receiving objects: 62% (156475/251390), 151.16 MiB | 722.00 KiB/s Receiving objects: 62% (157251/251390), 153.79 MiB | 723.00 KiB/s Receiving objects: 63% (158376/251390), 157.00 MiB | 766.00 KiB/s Receiving objects: 63% (158601/251390), 157.00 MiB | 766.00 KiB/s Receiving objects: 63% (159602/251390), 158.98 MiB | 705.00 KiB/s Receiving objects: 64% (160890/251390), 158.98 MiB | 705.00 KiB/s Receiving objects: 64% (162878/251390), 158.98 MiB | 705.00 KiB/s Receiving objects: 65% (163404/251390), 158.98 MiB | 705.00 KiB/s Receiving objects: 65% (164962/251390), 161.29 MiB | 1.44 MiB/s Receiving objects: 66% (165918/251390), 161.29 MiB | 1.44 MiB/s Receiving objects: 67% (168432/251390), 162.64 MiB | 1.44 MiB/s Receiving objects: 67% (170160/251390), 162.64 MiB | 1.44 MiB/s Receiving objects: 68% (170946/251390), 164.14 MiB | 1.49 MiB/s Receiving objects: 68% (173207/251390), 165.68 MiB | 572.00 KiB/s Receiving objects: 69% (173460/251390), 165.68 MiB | 572.00 KiB/s Receiving objects: 69% (173985/251390), 166.08 MiB | 304.00 KiB/s Receiving objects: 70% (175973/251390), 166.08 MiB | 304.00 KiB/s Receiving objects: 70% (177675/251390), 167.77 MiB | 255.00 KiB/s Receiving objects: 70% (177930/251390), 168.50 MiB | 226.00 KiB/s Receiving objects: 71% (178487/251390), 168.50 MiB | 226.00 KiB/s Receiving objects: 71% (179228/251390), 170.06 MiB | 273.00 KiB/s Receiving objects: 72% (181001/251390), 172.93 MiB | 274.00 KiB/s Receiving objects: 72% (181007/251390), 172.93 MiB | 274.00 KiB/s Receiving objects: 72% (181539/251390), 174.25 MiB | 391.00 KiB/s Receiving objects: 73% (183515/251390), 174.25 MiB | 391.00 KiB/s Receiving objects: 74% (186029/251390), 176.36 MiB | 888.00 KiB/s Receiving objects: 74% (187422/251390), 176.36 MiB | 888.00 KiB/s Receiving objects: 75% (188543/251390), 176.36 MiB | 888.00 KiB/s Receiving objects: 76% (191057/251390), 176.36 MiB | 888.00 KiB/s Receiving objects: 77% (193571/251390), 176.36 MiB | 888.00 KiB/s Receiving objects: 78% (196085/251390), 178.30 MiB | 2.33 MiB/s Receiving objects: 79% (198599/251390), 178.30 MiB | 2.33 MiB/s Receiving objects: 80% (201112/251390), 178.30 MiB | 2.33 MiB/s Receiving objects: 81% (203626/251390), 178.30 MiB | 2.33 MiB/s Receiving objects: 82% (206140/251390), 180.53 MiB | 2.67 MiB/s Receiving objects: 83% (208654/251390), 180.53 MiB | 2.67 MiB/s Receiving objects: 83% (209056/251390), 180.53 MiB | 2.67 MiB/s Receiving objects: 84% (211168/251390), 180.53 MiB | 2.67 MiB/s Receiving objects: 85% (213682/251390), 180.53 MiB | 2.67 MiB/s Receiving objects: 86% (216196/251390), 180.53 MiB | 2.67 MiB/s Receiving objects: 87% (218710/251390), 182.39 MiB | 2.73 MiB/s Receiving objects: 88% (221224/251390), 182.39 MiB | 2.73 MiB/s Receiving objects: 89% (223738/251390), 182.39 MiB | 2.73 MiB/s Receiving objects: 90% (226251/251390), 182.39 MiB | 2.73 MiB/s Receiving objects: 90% (227867/251390), 184.66 MiB | 2.90 MiB/s Receiving objects: 91% (228765/251390), 186.82 MiB | 3.08 MiB/s Receiving objects: 92% (231279/251390), 189.00 MiB | 3.38 MiB/s Receiving objects: 93% (233793/251390), 189.00 MiB | 3.38 MiB/s Receiving objects: 93% (234728/251390), 189.00 MiB | 3.38 MiB/s Receiving objects: 94% (236307/251390), 189.00 MiB | 3.38 MiB/s Receiving objects: 95% (238821/251390), 189.00 MiB | 3.38 MiB/s Receiving objects: 95% (239396/251390), 194.12 MiB | 3.95 MiB/s Receiving objects: 95% (239396/251390), 197.40 MiB | 3.75 MiB/s Receiving objects: 95% (239396/251390), 200.95 MiB | 3.62 MiB/s Receiving objects: 95% (239396/251390), 203.76 MiB | 3.28 MiB/s Receiving objects: 95% (239396/251390), 206.84 MiB | 3.14 MiB/s Receiving objects: 95% (239396/251390), 210.93 MiB | 3.37 MiB/s Receiving objects: 95% (239396/251390), 216.62 MiB | 3.89 MiB/s Receiving objects: 95% (239397/251390), 219.22 MiB | 4.04 MiB/s Receiving objects: 95% (239397/251390), 224.12 MiB | 4.16 MiB/s Receiving objects: 95% (239397/251390), 227.43 MiB | 4.19 MiB/s Receiving objects: 95% (239397/251390), 231.31 MiB | 3.88 MiB/s Receiving objects: 95% (239397/251390), 234.46 MiB | 3.39 MiB/s Receiving objects: 95% (239397/251390), 239.02 MiB | 3.71 MiB/s Receiving objects: 95% (239397/251390), 244.05 MiB | 4.04 MiB/s Receiving objects: 96% (241335/251390), 244.05 MiB | 4.04 MiB/s Receiving objects: 97% (243849/251390), 245.30 MiB | 3.97 MiB/s Receiving objects: 98% (246363/251390), 245.30 MiB | 3.97 MiB/s Receiving objects: 98% (246423/251390), 245.30 MiB | 3.97 MiB/s Receiving objects: 99% (248877/251390), 247.11 MiB | 3.94 MiB/s Receiving objects: 99% (249731/251390), 248.38 MiB | 3.79 MiB/s Receiving objects: 99% (250206/251390), 252.19 MiB | 3.93 MiB/s Receiving objects: 99% (250816/251390), 256.84 MiB | 3.95 MiB/s remote: Total 251390 (delta 7398), reused 10501 (delta 6596), pack-reused 239774 Receiving objects: 100% (251390/251390), 259.41 MiB | 3.82 MiB/s Receiving objects: 100% (251390/251390), 259.85 MiB | 1.68 MiB/s, done. Resolving deltas: 0% (0/173512) Resolving deltas: 0% (1178/173512) Resolving deltas: 0% (1439/173512) Resolving deltas: 0% (1714/173512) Resolving deltas: 1% (1736/173512) Resolving deltas: 1% (1957/173512) Resolving deltas: 1% (2175/173512) Resolving deltas: 1% (2346/173512) Resolving deltas: 1% (2517/173512) Resolving deltas: 1% (2706/173512) Resolving deltas: 1% (2926/173512) Resolving deltas: 1% (3161/173512) Resolving deltas: 1% (3462/173512) Resolving deltas: 2% (3471/173512) Resolving deltas: 2% (4000/173512) Resolving deltas: 2% (4180/173512) Resolving deltas: 2% (4897/173512) Resolving deltas: 2% (4997/173512) Resolving deltas: 2% (5085/173512) Resolving deltas: 2% (5186/173512) Resolving deltas: 3% (5206/173512) Resolving deltas: 3% (5300/173512) Resolving deltas: 3% (5511/173512) Resolving deltas: 4% (6941/173512) Resolving deltas: 4% (7267/173512) Resolving deltas: 5% (8676/173512) Resolving deltas: 5% (9777/173512) Resolving deltas: 5% (10027/173512) Resolving deltas: 5% (10351/173512) Resolving deltas: 6% (10411/173512) Resolving deltas: 6% (11243/173512) Resolving deltas: 6% (11740/173512) Resolving deltas: 6% (11922/173512) Resolving deltas: 7% (12146/173512) Resolving deltas: 7% (13323/173512) Resolving deltas: 8% (13881/173512) Resolving deltas: 9% (15617/173512) Resolving deltas: 10% (17352/173512) Resolving deltas: 11% (19087/173512) Resolving deltas: 11% (19332/173512) Resolving deltas: 11% (20277/173512) Resolving deltas: 12% (20822/173512) Resolving deltas: 12% (22364/173512) Resolving deltas: 13% (22557/173512) Resolving deltas: 13% (23410/173512) Resolving deltas: 14% (24292/173512) Resolving deltas: 14% (25736/173512) Resolving deltas: 15% (26027/173512) Resolving deltas: 15% (26924/173512) Resolving deltas: 16% (27762/173512) Resolving deltas: 16% (28469/173512) Resolving deltas: 16% (29200/173512) Resolving deltas: 17% (29498/173512) Resolving deltas: 18% (31233/173512) Resolving deltas: 18% (32393/173512) Resolving deltas: 19% (32968/173512) Resolving deltas: 19% (33036/173512) Resolving deltas: 19% (34098/173512) Resolving deltas: 20% (34703/173512) Resolving deltas: 20% (36070/173512) Resolving deltas: 21% (36438/173512) Resolving deltas: 22% (38173/173512) Resolving deltas: 22% (39471/173512) Resolving deltas: 23% (39908/173512) Resolving deltas: 23% (41015/173512) Resolving deltas: 24% (41643/173512) Resolving deltas: 25% (43378/173512) Resolving deltas: 25% (43719/173512) Resolving deltas: 26% (45114/173512) Resolving deltas: 26% (45568/173512) Resolving deltas: 26% (46031/173512) Resolving deltas: 27% (46849/173512) Resolving deltas: 28% (48584/173512) Resolving deltas: 28% (49573/173512) Resolving deltas: 29% (50319/173512) Resolving deltas: 29% (51973/173512) Resolving deltas: 30% (52054/173512) Resolving deltas: 31% (53789/173512) Resolving deltas: 31% (54440/173512) Resolving deltas: 32% (55525/173512) Resolving deltas: 32% (57144/173512) Resolving deltas: 33% (57259/173512) Resolving deltas: 33% (58386/173512) Resolving deltas: 34% (58995/173512) Resolving deltas: 34% (59989/173512) Resolving deltas: 35% (60730/173512) Resolving deltas: 36% (62465/173512) Resolving deltas: 37% (64200/173512) Resolving deltas: 37% (65881/173512) Resolving deltas: 38% (65935/173512) Resolving deltas: 39% (67670/173512) Resolving deltas: 40% (69405/173512) Resolving deltas: 41% (71140/173512) Resolving deltas: 41% (72597/173512) Resolving deltas: 42% (72876/173512) Resolving deltas: 43% (74611/173512) Resolving deltas: 44% (76346/173512) Resolving deltas: 45% (78081/173512) Resolving deltas: 46% (79816/173512) Resolving deltas: 47% (81551/173512) Resolving deltas: 47% (83207/173512) Resolving deltas: 48% (83286/173512) Resolving deltas: 49% (85021/173512) Resolving deltas: 49% (86674/173512) Resolving deltas: 50% (86756/173512) Resolving deltas: 51% (88492/173512) Resolving deltas: 52% (90227/173512) Resolving deltas: 53% (91962/173512) Resolving deltas: 53% (92633/173512) Resolving deltas: 54% (93697/173512) Resolving deltas: 55% (95432/173512) Resolving deltas: 56% (97167/173512) Resolving deltas: 57% (98902/173512) Resolving deltas: 58% (100637/173512) Resolving deltas: 59% (102373/173512) Resolving deltas: 59% (103550/173512) Resolving deltas: 60% (104108/173512) Resolving deltas: 61% (105843/173512) Resolving deltas: 62% (107578/173512) Resolving deltas: 63% (109313/173512) Resolving deltas: 64% (111048/173512) Resolving deltas: 65% (112783/173512) Resolving deltas: 66% (114518/173512) Resolving deltas: 66% (115838/173512) Resolving deltas: 67% (116254/173512) Resolving deltas: 68% (117989/173512) Resolving deltas: 69% (119724/173512) Resolving deltas: 69% (120824/173512) Resolving deltas: 70% (121459/173512) Resolving deltas: 71% (123194/173512) Resolving deltas: 71% (123493/173512) Resolving deltas: 72% (124929/173512) Resolving deltas: 72% (125784/173512) Resolving deltas: 73% (126664/173512) Resolving deltas: 74% (128399/173512) Resolving deltas: 75% (130134/173512) Resolving deltas: 75% (131195/173512) Resolving deltas: 76% (131870/173512) Resolving deltas: 77% (133605/173512) Resolving deltas: 77% (134201/173512) Resolving deltas: 78% (135340/173512) Resolving deltas: 78% (136198/173512) Resolving deltas: 79% (137075/173512) Resolving deltas: 79% (138071/173512) Resolving deltas: 80% (138810/173512) Resolving deltas: 80% (138982/173512) Resolving deltas: 80% (139057/173512) Resolving deltas: 80% (139497/173512) Resolving deltas: 80% (140093/173512) Resolving deltas: 81% (140545/173512) Resolving deltas: 82% (142280/173512) Resolving deltas: 83% (144015/173512) Resolving deltas: 84% (145751/173512) Resolving deltas: 85% (147486/173512) Resolving deltas: 86% (149221/173512) Resolving deltas: 86% (149228/173512) Resolving deltas: 87% (150956/173512) Resolving deltas: 88% (152691/173512) Resolving deltas: 89% (154426/173512) Resolving deltas: 89% (154742/173512) Resolving deltas: 90% (156161/173512) Resolving deltas: 91% (157896/173512) Resolving deltas: 91% (159500/173512) Resolving deltas: 92% (159632/173512) Resolving deltas: 93% (161367/173512) Resolving deltas: 93% (162563/173512) Resolving deltas: 94% (163102/173512) Resolving deltas: 95% (164837/173512) Resolving deltas: 95% (165501/173512) Resolving deltas: 96% (166572/173512) Resolving deltas: 97% (168307/173512) Resolving deltas: 98% (170042/173512) Resolving deltas: 98% (170867/173512) Resolving deltas: 98% (171145/173512) Resolving deltas: 98% (171719/173512) Resolving deltas: 99% (171777/173512) Resolving deltas: 99% (173438/173512) Resolving deltas: 100% (173512/173512) Resolving deltas: 100% (173512/173512), done.  -> Found systemd.service  -> Found sysusers.conf  -> Found tmpfiles.conf  -> Found config.yaml ==> Validating source files with b2sums... mimir ... Skipped systemd.service ... Passed sysusers.conf ... Passed tmpfiles.conf ... Passed config.yaml ... Passed ==> Making package: mimir 2.8.0-1 (Thu May 11 18:21:25 2023) ==> Checking runtime dependencies... ==> Checking buildtime dependencies... ==> Installing missing dependencies... [?25lresolving dependencies... looking for conflicting packages... Package (5) New Version Net Change extra/perl-error 0.17029-4 0.04 MiB extra/perl-mailtools 2.21-6 0.11 MiB extra/perl-timedate 2.33-4 0.08 MiB extra/git 2.40.1-1 33.31 MiB community/go 2:1.20.2-1 195.06 MiB Total Installed Size: 228.61 MiB :: Proceed with installation? [Y/n] checking keyring... checking package integrity... loading package files... checking for file conflicts... :: Processing package changes... installing perl-error... installing perl-timedate... installing perl-mailtools... installing git... Optional dependencies for git tk: gitk and git gui perl-libwww: git svn perl-term-readkey: git svn and interactive.singlekey setting perl-io-socket-ssl: git send-email TLS support perl-authen-sasl: git send-email TLS support perl-mediawiki-api: git mediawiki support perl-datetime-format-iso8601: git mediawiki support perl-lwp-protocol-https: git mediawiki https support perl-cgi: gitweb (web interface) support python: git svn & git p4 subversion: git svn org.freedesktop.secrets: keyring credential helper libsecret: libsecret credential helper [installed] installing go... :: Running post-transaction hooks... (1/1) Warn about old perl modules [?25h==> Retrieving sources...  -> Found systemd.service  -> Found sysusers.conf  -> Found tmpfiles.conf  -> Found config.yaml ==> WARNING: Skipping all source file integrity checks. ==> Extracting sources...  -> Creating working copy of mimir git repo... Cloning into 'mimir'... done. Updating files: 12% (1133/9223) Updating files: 13% (1199/9223) Updating files: 14% (1292/9223) Updating files: 15% (1384/9223) Updating files: 16% (1476/9223) Updating files: 17% (1568/9223) Updating files: 18% (1661/9223) Updating files: 19% (1753/9223) Updating files: 20% (1845/9223) Updating files: 21% (1937/9223) Updating files: 22% (2030/9223) Updating files: 23% (2122/9223) Updating files: 24% (2214/9223) Updating files: 25% (2306/9223) Updating files: 25% (2384/9223) Updating files: 26% (2398/9223) Updating files: 27% (2491/9223) Updating files: 28% (2583/9223) Updating files: 29% (2675/9223) Updating files: 30% (2767/9223) Updating files: 31% (2860/9223) Updating files: 32% (2952/9223) Updating files: 33% (3044/9223) Updating files: 34% (3136/9223) Updating files: 35% (3229/9223) Updating files: 35% (3259/9223) Updating files: 36% (3321/9223) Updating files: 37% (3413/9223) Updating files: 38% (3505/9223) Updating files: 39% (3597/9223) Updating files: 40% (3690/9223) Updating files: 40% (3765/9223) Updating files: 41% (3782/9223) Updating files: 42% (3874/9223) Updating files: 43% (3966/9223) Updating files: 44% (4059/9223) Updating files: 45% (4151/9223) Updating files: 46% (4243/9223) Updating files: 47% (4335/9223) Updating files: 48% (4428/9223) Updating files: 48% (4454/9223) Updating files: 49% (4520/9223) Updating files: 50% (4612/9223) Updating files: 51% (4704/9223) Updating files: 52% (4796/9223) Updating files: 53% (4889/9223) Updating files: 54% (4981/9223) Updating files: 55% (5073/9223) Updating files: 56% (5165/9223) Updating files: 57% (5258/9223) Updating files: 58% (5350/9223) Updating files: 59% (5442/9223) Updating files: 60% (5534/9223) Updating files: 61% (5627/9223) Updating files: 62% (5719/9223) Updating files: 62% (5788/9223) Updating files: 63% (5811/9223) Updating files: 64% (5903/9223) Updating files: 65% (5995/9223) Updating files: 66% (6088/9223) Updating files: 67% (6180/9223) Updating files: 68% (6272/9223) Updating files: 69% (6364/9223) Updating files: 70% (6457/9223) Updating files: 71% (6549/9223) Updating files: 72% (6641/9223) Updating files: 73% (6733/9223) Updating files: 74% (6826/9223) Updating files: 75% (6918/9223) Updating files: 76% (7010/9223) Updating files: 76% (7055/9223) Updating files: 77% (7102/9223) Updating files: 78% (7194/9223) Updating files: 79% (7287/9223) Updating files: 80% (7379/9223) Updating files: 81% (7471/9223) Updating files: 82% (7563/9223) Updating files: 83% (7656/9223) Updating files: 84% (7748/9223) Updating files: 85% (7840/9223) Updating files: 86% (7932/9223) Updating files: 87% (8025/9223) Updating files: 88% (8117/9223) Updating files: 89% (8209/9223) Updating files: 90% (8301/9223) Updating files: 90% (8337/9223) Updating files: 91% (8393/9223) Updating files: 92% (8486/9223) Updating files: 93% (8578/9223) Updating files: 94% (8670/9223) Updating files: 95% (8762/9223) Updating files: 96% (8855/9223) Updating files: 97% (8947/9223) Updating files: 98% (9039/9223) Updating files: 99% (9131/9223) Updating files: 100% (9223/9223) Updating files: 100% (9223/9223), done. Updating files: 11% (108/937) Updating files: 12% (113/937) Updating files: 13% (122/937) Updating files: 13% (124/937) Updating files: 14% (132/937) Updating files: 15% (141/937) Updating files: 16% (150/937) Updating files: 17% (160/937) Updating files: 18% (169/937) Updating files: 19% (179/937) Updating files: 20% (188/937) Updating files: 21% (197/937) Updating files: 22% (207/937) Updating files: 23% (216/937) Updating files: 24% (225/937) Updating files: 25% (235/937) Updating files: 26% (244/937) Updating files: 27% (253/937) Updating files: 28% (263/937) Updating files: 29% (272/937) Updating files: 30% (282/937) Updating files: 31% (291/937) Updating files: 32% (300/937) Updating files: 33% (310/937) Updating files: 34% (319/937) Updating files: 35% (328/937) Updating files: 36% (338/937) Updating files: 37% (347/937) Updating files: 38% (357/937) Updating files: 39% (366/937) Updating files: 40% (375/937) Updating files: 41% (385/937) Updating files: 42% (394/937) Updating files: 43% (403/937) Updating files: 44% (413/937) Updating files: 45% (422/937) Updating files: 46% (432/937) Updating files: 47% (441/937) Updating files: 48% (450/937) Updating files: 48% (451/937) Updating files: 49% (460/937) Updating files: 50% (469/937) Updating files: 51% (478/937) Updating files: 52% (488/937) Updating files: 53% (497/937) Updating files: 54% (506/937) Updating files: 55% (516/937) Updating files: 56% (525/937) Updating files: 57% (535/937) Updating files: 58% (544/937) Updating files: 59% (553/937) Updating files: 60% (563/937) Updating files: 61% (572/937) Updating files: 62% (581/937) Updating files: 63% (591/937) Updating files: 64% (600/937) Updating files: 65% (610/937) Updating files: 66% (619/937) Updating files: 67% (628/937) Updating files: 68% (638/937) Updating files: 69% (647/937) Updating files: 70% (656/937) Updating files: 71% (666/937) Updating files: 72% (675/937) Updating files: 73% (685/937) Updating files: 74% (694/937) Updating files: 75% (703/937) Updating files: 76% (713/937) Updating files: 77% (722/937) Updating files: 78% (731/937) Updating files: 79% (741/937) Updating files: 80% (750/937) Updating files: 81% (759/937) Updating files: 82% (769/937) Updating files: 83% (778/937) Updating files: 84% (788/937) Updating files: 85% (797/937) Updating files: 86% (806/937) Updating files: 87% (816/937) Updating files: 88% (825/937) Updating files: 89% (834/937) Updating files: 90% (844/937) Updating files: 91% (853/937) Updating files: 92% (863/937) Updating files: 93% (872/937) Updating files: 94% (881/937) Updating files: 95% (891/937) Updating files: 96% (900/937) Updating files: 97% (909/937) Updating files: 98% (919/937) Updating files: 99% (928/937) Updating files: 100% (937/937) Updating files: 100% (937/937), done. Switched to a new branch 'makepkg' ==> Starting prepare()... ==> Starting pkgver()... ==> Starting build()... all modules verified go: downloading github.com/go-kit/log v0.2.1 go: downloading github.com/grafana/dskit v0.0.0-20230413140051-dcfeb210d90e go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/prometheus/client_golang v1.14.0 go: downloading github.com/weaveworks/common v0.0.0-20230119144549-0aaa5abd1e63 go: downloading github.com/colega/go-yaml-yaml v0.0.0-20220720105220-255a8d16d094 go: downloading gopkg.in/alecthomas/kingpin.v2 v2.2.6 go: downloading github.com/thanos-io/objstore v0.0.0-20230201072718-11ffbc490204 go: downloading golang.org/x/exp v0.0.0-20230321023759-10a507213a29 go: downloading github.com/opentracing-contrib/go-stdlib v1.0.0 go: downloading github.com/opentracing/opentracing-go v1.2.1-0.20220228012449-10b1cf09e00b go: downloading github.com/grafana/mimir-prometheus v0.0.0-20230417132058-c461e223418b go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading go.opentelemetry.io/otel v1.14.0 go: downloading go.opentelemetry.io/otel/trace v1.14.0 go: downloading go.uber.org/atomic v1.10.0 go: downloading google.golang.org/grpc v1.47.0 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/cespare/xxhash/v2 v2.2.0 go: downloading github.com/golang/protobuf v1.5.3 go: downloading github.com/cespare/xxhash v1.1.0 go: downloading github.com/prometheus/client_model v0.3.0 go: downloading github.com/prometheus/common v0.42.0 go: downloading github.com/prometheus/procfs v0.9.0 go: downloading google.golang.org/protobuf v1.29.1 go: downloading github.com/alecthomas/units v0.0.0-20211218093645-b94a6e3cc137 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/golang/snappy v0.0.4 go: downloading golang.org/x/sync v0.1.0 go: downloading golang.org/x/time v0.3.0 go: downloading github.com/gorilla/mux v1.8.0 go: downloading github.com/sirupsen/logrus v1.9.0 go: downloading github.com/weaveworks/promrus v1.2.0 go: downloading github.com/opentracing-contrib/go-grpc v0.0.0-20210225150812-73cb765af46e go: downloading github.com/prometheus/exporter-toolkit v0.9.1 go: downloading golang.org/x/net v0.8.0 go: downloading github.com/grafana-tools/sdk v0.0.0-20211220201350-966b3088eec9 go: downloading github.com/prometheus/alertmanager v0.25.1-0.20230119163903-f59460bfd4bf go: downloading github.com/google/go-github/v32 v32.1.0 go: downloading github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751 go: downloading github.com/go-logfmt/logfmt v0.6.0 go: downloading github.com/grafana/regexp v0.0.0-20221005093135-b4c2bcb0a4b6 go: downloading github.com/minio/minio-go/v7 v7.0.45 go: downloading github.com/stretchr/testify v1.8.2 go: downloading github.com/oklog/ulid v1.3.1 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/efficientgo/core v1.0.0-rc.0.0.20221201130417-ba593f67d2a4 go: downloading github.com/gogo/status v1.1.1 go: downloading github.com/armon/go-metrics v0.4.1 go: downloading github.com/hashicorp/go-sockaddr v1.0.2 go: downloading github.com/grafana/memberlist v0.3.1-0.20220714140823-09ffed8adbbe go: downloading github.com/felixge/fgprof v0.9.2 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/charleskorn/goautoneg v0.0.0-20230303030534-7248a2f4c9cc go: downloading github.com/segmentio/fasthash v0.0.0-20180216231524-a72b379d632e go: downloading github.com/spf13/afero v1.8.2 go: downloading github.com/google/go-cmp v0.5.9 go: downloading github.com/dennwc/varint v1.0.0 go: downloading github.com/google/uuid v1.3.0 go: downloading github.com/edsrzf/mmap-go v1.1.0 go: downloading github.com/hashicorp/vault/api v1.9.0 go: downloading github.com/DmitriyVTitov/size v1.5.0 go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading github.com/prometheus/common/sigv4 v0.1.0 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.40.0 go: downloading github.com/sercand/kuberesolver v2.4.0+incompatible go: downloading github.com/go-logr/logr v1.2.3 go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.4 go: downloading golang.org/x/sys v0.6.0 go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading golang.org/x/crypto v0.7.0 go: downloading github.com/gogo/googleapis v1.4.1 go: downloading github.com/felixge/httpsnoop v1.0.3 go: downloading google.golang.org/genproto v0.0.0-20230306155012-7f2fa6fef1f4 go: downloading github.com/gosimple/slug v1.1.1 go: downloading github.com/alecthomas/chroma v0.10.0 go: downloading github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db go: downloading github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f go: downloading golang.org/x/oauth2 v0.6.0 go: downloading github.com/google/go-querystring v1.1.0 go: downloading github.com/aws/aws-sdk-go v1.44.217 go: downloading github.com/aws/aws-sdk-go-v2 v1.16.0 go: downloading github.com/aws/aws-sdk-go-v2/config v1.15.1 go: downloading github.com/facette/natsort v0.0.0-20181210072756-2cd4dd1e2dcb go: downloading github.com/go-redis/redis/v8 v8.11.5 go: downloading github.com/grafana/gomemcache v0.0.0-20230316202710-a081dae0aba9 go: downloading github.com/hashicorp/golang-lru v0.6.0 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading github.com/stretchr/objx v0.5.0 go: downloading github.com/miekg/dns v1.1.53 go: downloading github.com/hashicorp/consul/api v1.20.0 go: downloading github.com/hashicorp/go-cleanhttp v0.5.2 go: downloading go.etcd.io/etcd/api/v3 v3.5.4 go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.4 go: downloading go.etcd.io/etcd/client/v3 v3.5.4 go: downloading github.com/hashicorp/go-immutable-radix v1.3.1 go: downloading github.com/go-openapi/swag v0.22.3 go: downloading github.com/hashicorp/golang-lru/v2 v2.0.2 go: downloading github.com/oklog/run v1.1.0 go: downloading github.com/benbjohnson/clock v1.3.0 go: downloading github.com/cenkalti/backoff/v4 v4.2.0 go: downloading gopkg.in/telebot.v3 v3.1.2 go: downloading github.com/gofrs/uuid v4.3.1+incompatible go: downloading golang.org/x/text v0.8.0 go: downloading github.com/julienschmidt/httprouter v1.3.0 go: downloading github.com/google/btree v1.0.1 go: downloading github.com/hashicorp/go-msgpack v1.1.5 go: downloading github.com/hashicorp/go-multierror v1.1.1 go: downloading github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529 go: downloading github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/prometheusremotewrite v0.73.0 go: downloading go.opentelemetry.io/collector/pdata v1.0.0-rc7 go: downloading go.uber.org/multierr v1.9.0 go: downloading github.com/google/pprof v0.0.0-20230228050547-1710fef4ab10 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/go-openapi/strfmt v0.21.3 go: downloading github.com/dustin/go-humanize v1.0.0 go: downloading github.com/cenkalti/backoff/v3 v3.2.2 go: downloading github.com/hashicorp/errwrap v1.1.0 go: downloading github.com/hashicorp/go-retryablehttp v0.7.2 go: downloading github.com/hashicorp/go-rootcerts v1.0.2 go: downloading github.com/hashicorp/go-secure-stdlib/parseutil v0.1.7 go: downloading github.com/hashicorp/go-secure-stdlib/strutil v0.1.2 go: downloading github.com/hashicorp/hcl v1.0.1-vault-5 go: downloading github.com/mitchellh/mapstructure v1.5.0 go: downloading gopkg.in/square/go-jose.v2 v2.6.0 go: downloading go.uber.org/goleak v1.2.1 go: downloading go.opentelemetry.io/otel/metric v0.37.0 go: downloading github.com/go-logr/stdr v1.2.2 go: downloading github.com/rainycape/unidecode v0.0.0-20150907023854-cb7f23ec59be go: downloading github.com/jpillora/backoff v1.0.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.3.1 go: downloading github.com/Azure/azure-sdk-for-go v67.2.0+incompatible go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.2.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v0.5.1 go: downloading cloud.google.com/go/storage v1.28.1 go: downloading google.golang.org/api v0.114.0 go: downloading cloud.google.com/go v0.110.0 go: downloading github.com/ncw/swift v1.0.53 go: downloading github.com/aws/smithy-go v1.11.1 go: downloading github.com/aws/aws-sdk-go-v2/credentials v1.11.0 go: downloading github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.12.1 go: downloading github.com/aws/aws-sdk-go-v2/internal/ini v1.3.8 go: downloading github.com/aws/aws-sdk-go-v2/service/sso v1.11.1 go: downloading github.com/aws/aws-sdk-go-v2/service/sts v1.16.1 go: downloading github.com/klauspost/compress v1.15.15 go: downloading github.com/minio/md5-simd v1.1.2 go: downloading github.com/minio/sha256-simd v1.0.0 go: downloading gopkg.in/ini.v1 v1.67.0 go: downloading github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f go: downloading github.com/hashicorp/go-hclog v1.4.0 go: downloading github.com/hashicorp/serf v0.10.1 go: downloading go.uber.org/zap v1.21.0 go: downloading github.com/mailru/easyjson v0.7.7 go: downloading github.com/go-openapi/analysis v0.21.4 go: downloading github.com/go-openapi/loads v0.21.2 go: downloading github.com/go-openapi/runtime v0.25.0 go: downloading github.com/rs/cors v1.8.3 go: downloading github.com/go-openapi/errors v0.20.3 go: downloading github.com/go-openapi/validate v0.22.1 go: downloading github.com/shurcooL/vfsgen v0.0.0-20200824052919-0d455de96546 go: downloading github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/prometheus v0.73.0 go: downloading go.opentelemetry.io/collector/semconv v0.73.0 go: downloading github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 go: downloading go.mongodb.org/mongo-driver v1.11.2 go: downloading github.com/ryanuber/go-glob v1.0.0 go: downloading github.com/golang/glog v1.0.0 go: downloading github.com/dlclark/regexp2 v1.4.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.1.2 go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v0.8.1 go: downloading cloud.google.com/go/compute/metadata v0.2.3 go: downloading cloud.google.com/go/compute v1.18.0 go: downloading cloud.google.com/go/iam v0.12.0 go: downloading github.com/googleapis/gax-go/v2 v2.7.1 go: downloading github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.7 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.1 go: downloading github.com/rs/xid v1.4.0 go: downloading github.com/klauspost/cpuid/v2 v2.1.1 go: downloading github.com/fatih/color v1.14.1 go: downloading github.com/mattn/go-isatty v0.0.17 go: downloading github.com/coreos/go-semver v0.3.0 go: downloading github.com/josharian/intern v1.0.0 go: downloading github.com/go-openapi/jsonpointer v0.19.6 go: downloading github.com/go-openapi/spec v0.20.8 go: downloading github.com/jessevdk/go-flags v1.5.0 go: downloading github.com/jmespath/go-jmespath v0.4.0 go: downloading github.com/shurcooL/httpfs v0.0.0-20190707220628-8d4bc4ba7749 go: downloading go.opentelemetry.io/collector/featuregate v0.73.0 go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 go: downloading go.opencensus.io v0.24.0 go: downloading golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 go: downloading github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.1 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/go-openapi/jsonreference v0.20.2 go: downloading github.com/golang-jwt/jwt/v4 v4.5.0 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.2.3 internal/goarch internal/unsafeheader internal/abi internal/cpu internal/coverage/rtcov internal/bytealg internal/goexperiment internal/goos runtime/internal/atomic runtime/internal/math runtime/internal/sys runtime/internal/syscall internal/race sync/atomic runtime unicode unicode/utf8 internal/itoa encoding math/bits math unicode/utf16 container/list crypto/internal/alias crypto/internal/boring/sig vendor/golang.org/x/crypto/cryptobyte/asn1 internal/nettrace vendor/golang.org/x/crypto/internal/alias google.golang.org/protobuf/internal/flags google.golang.org/protobuf/internal/set github.com/Azure/azure-sdk-for-go/sdk/azcore/cloud github.com/Azure/azure-sdk-for-go/sdk/internal/errorinfo github.com/Azure/azure-sdk-for-go/sdk/azcore/to github.com/AzureAD/microsoft-authentication-library-for-go/apps/cache github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/exported github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops/internal/grant github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/version google.golang.org/grpc/attributes google.golang.org/grpc/serviceconfig github.com/googleapis/gax-go/v2/internal github.com/golang/groupcache/lru go.opencensus.io go.opencensus.io/trace/internal golang.org/x/xerrors/internal cloud.google.com/go/storage/internal go.opencensus.io/internal/tagencoding github.com/google/go-cmp/cmp/internal/flags github.com/aws/aws-sdk-go/aws/client/metadata github.com/aws/aws-sdk-go/internal/sdkmath github.com/uber/jaeger-client-go/internal/baggage github.com/uber/jaeger-client-go/internal/reporterstats github.com/uber/jaeger-client-go/internal/throttler github.com/weaveworks/common/errors golang.org/x/net/internal/iana github.com/grafana/dskit/internal/math github.com/grafana/dskit/internal/slices golang.org/x/exp/constraints github.com/grafana/dskit/loser golang.org/x/exp/slices github.com/aws/aws-sdk-go-v2/internal/sdkio github.com/aws/aws-sdk-go-v2 github.com/dgryski/go-rendezvous github.com/prometheus/prometheus/model/value github.com/dennwc/varint github.com/prometheus/prometheus/tsdb/goversion go.opentelemetry.io/otel/internal go.mongodb.org/mongo-driver/bson/bsonoptions go.mongodb.org/mongo-driver/bson/bsontype go.opentelemetry.io/collector/semconv/v1.6.1 github.com/segmentio/fasthash/fnv1a image/color internal/reflectlite sync internal/testlog crypto/subtle internal/godebug math/rand internal/singleflight internal/intern runtime/cgo errors io bytes internal/oserror syscall strconv reflect time context sort internal/syscall/unix internal/poll internal/safefilepath internal/syscall/execenv path io/fs internal/fmtsort os strings encoding/binary fmt encoding/base64 bufio regexp/syntax flag encoding/json github.com/go-logfmt/logfmt log regexp github.com/alecthomas/units github.com/go-kit/log github.com/pkg/errors compress/flate github.com/go-kit/log/level hash hash/crc32 crypto crypto/cipher crypto/internal/boring compress/gzip crypto/internal/randutil math/big crypto/aes crypto/des crypto/internal/edwards25519/field crypto/internal/nistec/fiat crypto/rand crypto/internal/bigmod crypto/internal/boring/bbig crypto/sha512 encoding/asn1 crypto/internal/nistec vendor/golang.org/x/crypto/cryptobyte crypto/ecdh crypto/elliptic crypto/internal/edwards25519 crypto/ecdsa crypto/ed25519 crypto/hmac crypto/md5 crypto/rc4 crypto/rsa crypto/sha1 crypto/sha256 crypto/dsa encoding/hex encoding/pem crypto/x509/pkix vendor/golang.org/x/net/dns/dnsmessage net/netip net/url net path/filepath vendor/golang.org/x/crypto/chacha20 vendor/golang.org/x/crypto/internal/poly1305 vendor/golang.org/x/crypto/chacha20poly1305 vendor/golang.org/x/crypto/hkdf vendor/golang.org/x/text/transform vendor/golang.org/x/text/unicode/bidi vendor/golang.org/x/text/secure/bidirule vendor/golang.org/x/text/unicode/norm vendor/golang.org/x/net/idna vendor/golang.org/x/net/http2/hpack mime mime/quotedprintable net/http/internal net/http/internal/ascii github.com/beorn7/perks/quantile github.com/cespare/xxhash/v2 hash/fnv crypto/x509 net/textproto vendor/golang.org/x/net/http/httpguts vendor/golang.org/x/net/http/httpproxy mime/multipart google.golang.org/protobuf/internal/detrand google.golang.org/protobuf/internal/errors google.golang.org/protobuf/encoding/protowire google.golang.org/protobuf/internal/pragma google.golang.org/protobuf/reflect/protoreflect crypto/tls google.golang.org/protobuf/internal/encoding/messageset go/token google.golang.org/protobuf/internal/strs google.golang.org/protobuf/internal/encoding/text google.golang.org/protobuf/internal/genid google.golang.org/protobuf/internal/order google.golang.org/protobuf/reflect/protoregistry google.golang.org/protobuf/runtime/protoiface google.golang.org/protobuf/proto google.golang.org/protobuf/encoding/prototext google.golang.org/protobuf/internal/encoding/defval google.golang.org/protobuf/internal/descfmt net/http/httptrace google.golang.org/protobuf/internal/descopts google.golang.org/protobuf/internal/filedesc net/http google.golang.org/protobuf/internal/encoding/tag io/ioutil google.golang.org/protobuf/internal/impl google.golang.org/protobuf/internal/filetype google.golang.org/protobuf/internal/version google.golang.org/protobuf/runtime/protoimpl expvar google.golang.org/protobuf/types/descriptorpb google.golang.org/protobuf/types/known/timestamppb github.com/golang/protobuf/ptypes/timestamp github.com/prometheus/common/model google.golang.org/protobuf/reflect/protodesc runtime/metrics github.com/prometheus/common/internal/bitbucket.org/ww/goautoneg github.com/prometheus/procfs/internal/fs github.com/prometheus/procfs/internal/util golang.org/x/sys/unix github.com/golang/protobuf/proto github.com/prometheus/client_model/go github.com/prometheus/client_golang/prometheus/internal github.com/matttproud/golang_protobuf_extensions/pbutil github.com/prometheus/common/expfmt github.com/prometheus/procfs runtime/debug github.com/efficientgo/core/errors github.com/efficientgo/core/logerrcapture github.com/davecgh/go-spew/spew github.com/efficientgo/core/testutil/internal internal/sysinfo runtime/trace testing github.com/prometheus/client_golang/prometheus github.com/efficientgo/core/testutil github.com/opentracing/opentracing-go/log github.com/opentracing/opentracing-go github.com/thanos-io/objstore/tracing golang.org/x/sync/errgroup github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared golang.org/x/text/transform golang.org/x/text/unicode/bidi github.com/prometheus/client_golang/prometheus/promauto golang.org/x/text/secure/bidirule golang.org/x/text/unicode/norm github.com/grafana/dskit/flagext github.com/thanos-io/objstore github.com/Azure/azure-sdk-for-go/sdk/azcore/tracing encoding/xml golang.org/x/net/idna github.com/Azure/azure-sdk-for-go/sdk/internal/log github.com/Azure/azure-sdk-for-go/sdk/azcore/log github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/log github.com/Azure/azure-sdk-for-go/sdk/internal/diag golang.org/x/net/http/httpguts github.com/Azure/azure-sdk-for-go/sdk/internal/temporal github.com/Azure/azure-sdk-for-go/sdk/internal/uuid github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/json github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported github.com/Azure/azure-sdk-for-go/sdk/azcore/policy github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/pollers github.com/Azure/azure-sdk-for-go/sdk/azcore/streaming github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/json/types/time github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/pollers/async github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/pollers/body github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/pollers/loc github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/pollers/op github.com/kylelemons/godebug/diff github.com/kylelemons/godebug/pretty github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime github.com/AzureAD/microsoft-authentication-library-for-go/apps/errors database/sql/driver github.com/Azure/azure-sdk-for-go/sdk/azcore github.com/google/uuid github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/shared github.com/golang-jwt/jwt/v4 github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops/authority github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops/wstrust/defs github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops/internal/comm github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops/wstrust github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/options github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops/accesstokens github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/local os/exec github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth github.com/pkg/browser golang.org/x/crypto/pkcs12/internal/rc2 golang.org/x/crypto/pkcs12 github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/base/internal/storage github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/generated github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/base github.com/AzureAD/microsoft-authentication-library-for-go/apps/confidential github.com/AzureAD/microsoft-authentication-library-for-go/apps/public github.com/Azure/azure-sdk-for-go/sdk/azidentity github.com/thanos-io/objstore/exthttp gopkg.in/yaml.v2 github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/shared github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/exported github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/base github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/sas github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/bloberror github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blob gopkg.in/yaml.v3 github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blockblob github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/appendblob github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/pageblob github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/container github.com/efficientgo/core/merrors github.com/efficientgo/core/errcapture github.com/thanos-io/objstore/providers/filesystem github.com/grafana/mimir/pkg/storage/bucket/filesystem cloud.google.com/go/compute/metadata github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/service google.golang.org/genproto/googleapis/api google.golang.org/protobuf/types/known/durationpb google.golang.org/genproto/googleapis/api/annotations github.com/Azure/azure-sdk-for-go/sdk/storage/azblob github.com/thanos-io/objstore/providers/azure google.golang.org/genproto/googleapis/type/expr golang.org/x/net/internal/timeseries github.com/grafana/mimir/pkg/storage/bucket/azure html text/template/parse text/tabwriter google.golang.org/grpc/backoff google.golang.org/grpc/internal/grpclog google.golang.org/grpc/grpclog google.golang.org/grpc/connectivity google.golang.org/grpc/internal/credentials google.golang.org/grpc/credentials google.golang.org/grpc/internal/channelz text/template google.golang.org/grpc/channelz google.golang.org/protobuf/internal/encoding/json google.golang.org/protobuf/encoding/protojson html/template github.com/golang/protobuf/jsonpb golang.org/x/net/trace google.golang.org/grpc/internal/pretty google.golang.org/grpc/resolver google.golang.org/grpc/metadata google.golang.org/grpc/internal google.golang.org/grpc/internal/grpcrand google.golang.org/grpc/balancer google.golang.org/grpc/codes google.golang.org/grpc/credentials/insecure google.golang.org/grpc/balancer/base google.golang.org/grpc/encoding google.golang.org/grpc/encoding/proto google.golang.org/grpc/internal/backoff google.golang.org/grpc/internal/balancerload google.golang.org/grpc/balancer/roundrobin google.golang.org/grpc/internal/balancer/gracefulswitch google.golang.org/protobuf/types/known/anypb github.com/golang/protobuf/ptypes/duration google.golang.org/grpc/binarylog/grpc_binarylog_v1 github.com/golang/protobuf/ptypes/any github.com/golang/protobuf/ptypes google.golang.org/grpc/internal/grpcutil google.golang.org/genproto/googleapis/rpc/status google.golang.org/grpc/internal/buffer google.golang.org/grpc/internal/grpcsync google.golang.org/grpc/internal/metadata google.golang.org/grpc/internal/status google.golang.org/grpc/internal/serviceconfig google.golang.org/grpc/internal/resolver google.golang.org/grpc/status google.golang.org/grpc/balancer/grpclb/state google.golang.org/grpc/internal/envconfig google.golang.org/grpc/internal/resolver/dns google.golang.org/grpc/internal/binarylog google.golang.org/grpc/internal/resolver/passthrough google.golang.org/grpc/internal/transport/networktype google.golang.org/grpc/internal/resolver/unix golang.org/x/net/http2/hpack google.golang.org/grpc/internal/syscall google.golang.org/grpc/keepalive google.golang.org/grpc/peer google.golang.org/grpc/stats golang.org/x/net/http2 google.golang.org/grpc/tap net/http/httputil google.golang.org/protobuf/types/known/fieldmaskpb google.golang.org/genproto/googleapis/rpc/code github.com/googleapis/gax-go/v2/apierror/internal/proto google.golang.org/api/internal/third_party/uritemplates google.golang.org/api/googleapi google.golang.org/genproto/googleapis/rpc/errdetails github.com/googleapis/gax-go/v2/apierror cloud.google.com/go/internal/optional go.opencensus.io/internal go.opencensus.io/trace/tracestate go.opencensus.io/trace golang.org/x/xerrors cloud.google.com/go/internal/trace google.golang.org/grpc/internal/transport cloud.google.com/go/internal/version google.golang.org/genproto/googleapis/type/date google.golang.org/protobuf/types/known/emptypb google.golang.org/api/iterator golang.org/x/oauth2/internal golang.org/x/oauth2 golang.org/x/oauth2/authhandler golang.org/x/oauth2/google/internal/externalaccount golang.org/x/oauth2/jws golang.org/x/oauth2/jwt os/user google.golang.org/grpc golang.org/x/oauth2/google internal/saferio encoding/gob cloud.google.com/go/iam/apiv1/iampb github.com/googleapis/gax-go/v2 google.golang.org/genproto/googleapis/iam/v1 cloud.google.com/go/iam cloud.google.com/go/internal cloud.google.com/go/storage/internal/apiv2/stubs github.com/googleapis/enterprise-certificate-proxy/client/util net/rpc github.com/googleapis/enterprise-certificate-proxy/client google.golang.org/api/internal/cert google.golang.org/api/internal/impersonate google.golang.org/api/internal google.golang.org/api/option google.golang.org/api/option/internaloption go.opencensus.io/resource go.opencensus.io/metric/metricdata runtime/pprof go.opencensus.io/tag go.opencensus.io/metric/metricproducer go.opencensus.io/stats/internal go.opencensus.io/trace/propagation go.opencensus.io/stats github.com/google/go-cmp/cmp/internal/diff go.opencensus.io/stats/view github.com/google/go-cmp/cmp/internal/function github.com/google/go-cmp/cmp/internal/value github.com/google/go-cmp/cmp go.opencensus.io/plugin/ocgrpc google.golang.org/grpc/balancer/grpclb/grpc_lb_v1 google.golang.org/grpc/credentials/alts/internal google.golang.org/grpc/credentials/alts/internal/proto/grpc_gcp google.golang.org/grpc/balancer/grpclb google.golang.org/grpc/credentials/alts/internal/authinfo google.golang.org/grpc/credentials/alts/internal/conn google.golang.org/grpc/credentials/alts/internal/handshaker google.golang.org/grpc/credentials/alts/internal/handshaker/service google.golang.org/grpc/internal/googlecloud google.golang.org/grpc/credentials/oauth google.golang.org/grpc/credentials/alts google.golang.org/api/internal/gensupport google.golang.org/grpc/credentials/google google.golang.org/api/transport/grpc go.opencensus.io/plugin/ochttp/propagation/b3 go.opencensus.io/plugin/ochttp cloud.google.com/go/storage/internal/apiv2 google.golang.org/api/googleapi/transport google.golang.org/api/transport/http/internal/propagation github.com/prometheus/common/version google.golang.org/api/transport/http github.com/aws/aws-sdk-go/aws/awserr github.com/aws/aws-sdk-go/internal/ini google.golang.org/api/iamcredentials/v1 google.golang.org/api/storage/v1 google.golang.org/api/transport github.com/aws/aws-sdk-go/internal/shareddefaults github.com/aws/aws-sdk-go/internal/sync/singleflight github.com/aws/aws-sdk-go/aws/credentials github.com/aws/aws-sdk-go/aws/endpoints cloud.google.com/go/storage github.com/thanos-io/objstore/providers/gcs github.com/grafana/mimir/pkg/storage/bucket/gcs github.com/aws/aws-sdk-go/internal/sdkio github.com/aws/aws-sdk-go/aws/arn github.com/jmespath/go-jmespath github.com/aws/aws-sdk-go/aws/awsutil github.com/aws/aws-sdk-go/internal/sdkrand github.com/aws/aws-sdk-go/internal/strings github.com/aws/aws-sdk-go/internal/s3shared/arn github.com/gogo/protobuf/proto github.com/aws/aws-sdk-go/aws github.com/aws/aws-sdk-go/aws/request github.com/aws/aws-sdk-go/aws/client github.com/aws/aws-sdk-go/private/protocol github.com/aws/aws-sdk-go/private/protocol/rest github.com/aws/aws-sdk-go/aws/signer/v4 github.com/aws/aws-sdk-go/internal/s3shared github.com/aws/aws-sdk-go/internal/s3shared/s3err github.com/aws/aws-sdk-go/private/checksum github.com/aws/aws-sdk-go/private/protocol/eventstream github.com/aws/aws-sdk-go/private/protocol/query/queryutil github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil github.com/aws/aws-sdk-go/private/protocol/eventstream/eventstreamapi github.com/aws/aws-sdk-go/private/protocol/query github.com/aws/aws-sdk-go/private/protocol/restxml github.com/golang/snappy github.com/aws/aws-sdk-go/service/s3 github.com/grafana/dskit/kv/codec github.com/grafana/dskit/backoff github.com/hashicorp/go-cleanhttp github.com/mattn/go-isatty github.com/mattn/go-colorable github.com/fatih/color github.com/hashicorp/go-hclog github.com/hashicorp/go-rootcerts github.com/hashicorp/golang-lru/simplelru github.com/hashicorp/go-immutable-radix os/signal github.com/armon/go-metrics github.com/hashicorp/serf/coordinate github.com/mitchellh/mapstructure github.com/hashicorp/consul/api github.com/opentracing/opentracing-go/ext github.com/weaveworks/common/grpc github.com/uber/jaeger-client-go/internal/spanlog github.com/uber/jaeger-client-go/log hash/adler32 compress/zlib github.com/uber/jaeger-client-go/thrift github.com/uber/jaeger-client-go/thrift-gen/jaeger github.com/uber/jaeger-client-go/thrift-gen/sampling github.com/uber/jaeger-client-go/thrift-gen/zipkincore github.com/uber/jaeger-lib/metrics go.uber.org/atomic github.com/uber/jaeger-client-go/thrift-gen/agent github.com/uber/jaeger-client-go/thrift-gen/baggage github.com/uber/jaeger-client-go/utils github.com/uber/jaeger-lib/metrics/prometheus github.com/uber/jaeger-client-go github.com/sirupsen/logrus github.com/weaveworks/promrus github.com/weaveworks/common/logging golang.org/x/net/context github.com/weaveworks/common/user golang.org/x/time/rate github.com/grafana/dskit/crypto/tls github.com/uber/jaeger-client-go/internal/baggage/remote github.com/uber/jaeger-client-go/internal/throttler/remote github.com/uber/jaeger-client-go/rpcmetrics github.com/uber/jaeger-client-go/transport github.com/gogo/protobuf/protoc-gen-gogo/descriptor github.com/uber/jaeger-client-go/config github.com/weaveworks/common/tracing github.com/weaveworks/common/instrument github.com/grafana/dskit/kv/consul github.com/gogo/protobuf/gogoproto go.uber.org/multierr go.uber.org/zap/buffer go.etcd.io/etcd/api/v3/authpb go.etcd.io/etcd/api/v3/membershippb go.etcd.io/etcd/api/v3/mvccpb go.uber.org/zap/internal/bufferpool go.uber.org/zap/internal/color go.uber.org/zap/internal/exit go.uber.org/zap/zapcore go.etcd.io/etcd/api/v3/etcdserverpb go.uber.org/zap go.etcd.io/etcd/client/pkg/v3/fileutil go.etcd.io/etcd/client/pkg/v3/tlsutil go.etcd.io/etcd/client/pkg/v3/transport go.etcd.io/etcd/api/v3/v3rpc/rpctypes github.com/coreos/go-semver/semver go.etcd.io/etcd/api/v3/version github.com/coreos/go-systemd/v22/journal go.etcd.io/etcd/client/pkg/v3/systemd go.etcd.io/etcd/client/pkg/v3/logutil go.etcd.io/etcd/client/pkg/v3/types go.etcd.io/etcd/client/v3/credentials go.etcd.io/etcd/client/v3/internal/endpoint google.golang.org/grpc/resolver/manual go.etcd.io/etcd/client/v3/internal/resolver go.uber.org/zap/zapgrpc embed github.com/prometheus/client_golang/prometheus/push github.com/armon/go-metrics/prometheus github.com/grafana/dskit/services github.com/hashicorp/go-sockaddr go.etcd.io/etcd/client/v3 compress/lzw github.com/google/btree github.com/hashicorp/go-msgpack/codec github.com/grafana/dskit/kv/etcd github.com/hashicorp/errwrap github.com/hashicorp/go-multierror encoding/base32 golang.org/x/net/bpf golang.org/x/net/internal/socket golang.org/x/net/ipv4 golang.org/x/net/ipv6 github.com/miekg/dns github.com/sean-/seed github.com/gogo/protobuf/sortkeys github.com/gogo/protobuf/types github.com/gogo/googleapis/google/rpc github.com/gogo/status github.com/weaveworks/common/httpgrpc github.com/grafana/dskit/netutil github.com/grafana/dskit/ring/shard github.com/grafana/dskit/tenant github.com/golang/protobuf/ptypes/empty github.com/gorilla/mux github.com/opentracing-contrib/go-grpc github.com/prometheus/client_golang/prometheus/promhttp github.com/coreos/go-systemd/v22/activation github.com/jpillora/backoff github.com/mwitkow/go-conntrack golang.org/x/net/http/httpproxy golang.org/x/oauth2/clientcredentials github.com/prometheus/common/config golang.org/x/crypto/blowfish golang.org/x/crypto/bcrypt github.com/prometheus/exporter-toolkit/web github.com/sercand/kuberesolver github.com/felixge/httpsnoop github.com/opentracing-contrib/go-stdlib/nethttp github.com/weaveworks/common/middleware net/http/internal/testcert net/http/httptest github.com/weaveworks/common/httpgrpc/server github.com/weaveworks/common/signals golang.org/x/net/netutil internal/profile net/http/pprof github.com/weaveworks/common/server github.com/grafana/mimir/pkg/util/log github.com/DmitriyVTitov/size github.com/dgraph-io/ristretto/z/simd github.com/dustin/go-humanize github.com/golang/glog github.com/dgraph-io/ristretto/z github.com/dgraph-io/ristretto github.com/grafana/regexp/syntax github.com/hashicorp/memberlist github.com/grafana/regexp github.com/prometheus/prometheus/model/labels github.com/grafana/dskit/kv/memberlist github.com/modern-go/concurrent github.com/modern-go/reflect2 github.com/grafana/dskit/kv github.com/json-iterator/go github.com/grafana/dskit/ring github.com/grafana/mimir/pkg/util golang.org/x/crypto/blake2b golang.org/x/crypto/argon2 github.com/minio/minio-go/v7/pkg/encrypt github.com/aws/aws-sdk-go-v2/internal/rand github.com/aws/aws-sdk-go-v2/internal/sdk github.com/aws/aws-sdk-go-v2/internal/sync/singleflight github.com/aws/smithy-go/logging github.com/aws/smithy-go/ptr github.com/aws/smithy-go/middleware github.com/aws/smithy-go/rand github.com/aws/smithy-go github.com/aws/smithy-go/time github.com/aws/aws-sdk-go-v2/aws github.com/aws/smithy-go/transport/http/internal/io github.com/aws/smithy-go/transport/http github.com/aws/aws-sdk-go-v2/aws/ratelimit github.com/aws/aws-sdk-go-v2/aws/middleware github.com/aws/aws-sdk-go-v2/credentials github.com/aws/aws-sdk-go-v2/internal/timeconv github.com/aws/aws-sdk-go-v2/feature/ec2/imds/internal/config github.com/aws/smithy-go/io github.com/aws/aws-sdk-go-v2/credentials/processcreds github.com/aws/aws-sdk-go-v2/aws/transport/http github.com/aws/aws-sdk-go-v2/aws/retry github.com/aws/aws-sdk-go-v2/aws/defaults github.com/aws/aws-sdk-go-v2/aws/protocol/restjson github.com/aws/aws-sdk-go-v2/internal/strings github.com/aws/aws-sdk-go-v2/aws/signer/internal/v4 github.com/aws/aws-sdk-go-v2/feature/ec2/imds github.com/aws/aws-sdk-go-v2/credentials/endpointcreds/internal/client github.com/aws/aws-sdk-go-v2/credentials/endpointcreds github.com/aws/aws-sdk-go-v2/credentials/ec2rolecreds github.com/aws/smithy-go/encoding/httpbinding github.com/aws/aws-sdk-go-v2/internal/configsources github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 github.com/aws/aws-sdk-go-v2/service/sso/internal/endpoints github.com/aws/aws-sdk-go-v2/aws/signer/v4 github.com/aws/smithy-go/document github.com/aws/aws-sdk-go-v2/service/sso/types github.com/aws/aws-sdk-go-v2/aws/protocol/query github.com/aws/aws-sdk-go-v2/service/sso github.com/aws/aws-sdk-go-v2/aws/protocol/xml github.com/aws/aws-sdk-go-v2/service/internal/presigned-url github.com/aws/aws-sdk-go-v2/service/sts/internal/endpoints github.com/aws/aws-sdk-go-v2/service/sts/types github.com/aws/smithy-go/encoding github.com/aws/smithy-go/encoding/xml github.com/aws/aws-sdk-go-v2/service/sts github.com/aws/aws-sdk-go-v2/credentials/ssocreds github.com/aws/aws-sdk-go-v2/internal/ini archive/tar github.com/klauspost/compress/s2 github.com/aws/aws-sdk-go-v2/credentials/stscreds github.com/aws/aws-sdk-go-v2/config github.com/minio/md5-simd github.com/minio/minio-go/v7/pkg/s3utils github.com/minio/minio-go/v7/pkg/signer gopkg.in/ini.v1 github.com/minio/minio-go/v7/pkg/lifecycle github.com/minio/minio-go/v7/pkg/set github.com/minio/minio-go/v7/pkg/notification github.com/minio/minio-go/v7/pkg/credentials github.com/rs/xid github.com/minio/minio-go/v7/pkg/replication github.com/minio/minio-go/v7/pkg/sse github.com/minio/minio-go/v7/pkg/tags github.com/klauspost/cpuid/v2 net/http/cookiejar golang.org/x/net/publicsuffix github.com/ncw/swift github.com/minio/sha256-simd github.com/minio/minio-go/v7 github.com/thanos-io/objstore/providers/swift github.com/grafana/mimir/pkg/storage/bucket/swift github.com/pmezard/go-difflib/difflib github.com/stretchr/objx github.com/stretchr/testify/assert github.com/stretchr/testify/mock github.com/thanos-io/objstore/providers/s3 github.com/cespare/xxhash github.com/grafana/mimir/pkg/storage/bucket/s3 github.com/facette/natsort github.com/go-redis/redis/v8/internal/rand github.com/go-redis/redis/v8/internal/util github.com/go-redis/redis/v8/internal github.com/grafana/mimir/pkg/storage/bucket github.com/go-redis/redis/v8/internal/hashtag github.com/go-redis/redis/v8/internal/hscan github.com/go-redis/redis/v8/internal/proto github.com/go-redis/redis/v8/internal/pool github.com/grafana/dskit/dns/godns github.com/go-redis/redis/v8 github.com/grafana/dskit/dns/miekgdns github.com/grafana/dskit/multierror github.com/grafana/dskit/dns github.com/grafana/dskit/gate github.com/grafana/dskit/promregistry github.com/grafana/dskit/spanlogger github.com/grafana/gomemcache/memcache github.com/grafana/dskit/grpcencoding/snappy google.golang.org/grpc/encoding/gzip github.com/grafana/dskit/grpcclient github.com/prometheus/prometheus/model/exemplar github.com/prometheus/prometheus/model/histogram github.com/prometheus/prometheus/prompb/io/prometheus/client github.com/prometheus/prometheus/model/textparse container/heap github.com/edsrzf/mmap-go github.com/prometheus/prometheus/model/timestamp github.com/prometheus/prometheus/model/metadata github.com/prometheus/prometheus/tsdb/chunkenc github.com/prometheus/prometheus/tsdb/errors github.com/prometheus/prometheus/tsdb/fileutil github.com/prometheus/prometheus/tsdb/chunks github.com/prometheus/client_golang/prometheus/testutil/promlint github.com/prometheus/client_golang/prometheus/testutil github.com/stretchr/testify/require go.uber.org/goleak/internal/stack go.uber.org/goleak github.com/prometheus/prometheus/util/testutil github.com/prometheus/prometheus/tsdb/tsdbutil github.com/prometheus/prometheus/storage github.com/prometheus/prometheus/util/strutil github.com/prometheus/prometheus/promql/parser github.com/oklog/ulid github.com/aws/aws-sdk-go/service/sts github.com/aws/aws-sdk-go/service/sts/stsiface github.com/aws/aws-sdk-go/aws/credentials/stscreds github.com/aws/aws-sdk-go/aws/corehandlers github.com/aws/aws-sdk-go/aws/credentials/processcreds github.com/aws/aws-sdk-go/private/protocol/json/jsonutil github.com/aws/aws-sdk-go/private/protocol/jsonrpc github.com/aws/aws-sdk-go/private/protocol/restjson github.com/aws/aws-sdk-go/service/sso github.com/aws/aws-sdk-go/service/sso/ssoiface github.com/aws/aws-sdk-go/aws/credentials/ssocreds github.com/aws/aws-sdk-go/aws/csm github.com/grafana/dskit/cache github.com/aws/aws-sdk-go/internal/sdkuri github.com/aws/aws-sdk-go/aws/ec2metadata github.com/aws/aws-sdk-go/aws/credentials/ec2rolecreds github.com/aws/aws-sdk-go/aws/credentials/endpointcreds github.com/aws/aws-sdk-go/aws/defaults github.com/aws/aws-sdk-go/aws/session github.com/prometheus/prometheus/discovery/targetgroup github.com/prometheus/prometheus/discovery github.com/prometheus/common/sigv4 github.com/prometheus/prometheus/model/relabel github.com/prometheus/prometheus/tsdb/encoding github.com/prometheus/prometheus/tsdb/hashcache github.com/prometheus/prometheus/tsdb/index github.com/prometheus/prometheus/config github.com/prometheus/prometheus/tsdb/tombstones github.com/prometheus/prometheus/tsdb/record github.com/prometheus/prometheus/tsdb/wlog github.com/prometheus/prometheus/util/zeropool golang.org/x/sync/semaphore github.com/go-logr/logr github.com/go-logr/logr/funcr github.com/go-logr/stdr go.opentelemetry.io/otel/internal/attribute github.com/prometheus/prometheus/tsdb go.opentelemetry.io/otel/attribute go.opentelemetry.io/otel/codes go.opentelemetry.io/otel/internal/baggage go.opentelemetry.io/otel/baggage go.opentelemetry.io/otel/trace go.opentelemetry.io/otel/propagation go.opentelemetry.io/otel/internal/global go.opentelemetry.io/otel github.com/prometheus/prometheus/util/stats github.com/prometheus/prometheus/util/jsonutil google.golang.org/grpc/health/grpc_health_v1 github.com/golang/groupcache/singleflight github.com/grafana/dskit/runutil github.com/grafana/mimir/pkg/util/extprom github.com/grafana/mimir/pkg/util/pool github.com/grafana/mimir/pkg/storage/tsdb/bucketcache github.com/grafana/mimir/pkg/storage/sharding github.com/grafana/mimir/pkg/util/spanlogger github.com/grafana/mimir/pkg/storegateway/indexcache github.com/grafana/mimir/pkg/util/math github.com/grafana/mimir/pkg/storegateway/indexheader/encoding github.com/grafana/mimir/pkg/storegateway/indexheader/index github.com/grafana/dskit/grpcutil github.com/prometheus/prometheus/util/teststorage github.com/grafana/mimir/pkg/storage/tsdb/metadata github.com/grafana/mimir/pkg/storage/tsdb/block github.com/prometheus/prometheus/promql github.com/grafana/mimir/pkg/storegateway/indexheader github.com/grafana/dskit/modules github.com/grafana/dskit/runtimeconfig github.com/grafana/dskit/concurrency github.com/grafana/dskit/metrics github.com/grafana/dskit/ring/client github.com/grafana/mimir/pkg/mimirpb github.com/prometheus/alertmanager/cluster/clusterpb github.com/grafana/mimir/pkg/alertmanager/alertmanagerpb github.com/grafana/mimir/pkg/alertmanager/alertspb github.com/grafana/mimir/pkg/alertmanager/alertstore/bucketclient github.com/prometheus/alertmanager/pkg/labels github.com/prometheus/alertmanager/timeinterval github.com/prometheus/alertmanager/config github.com/grafana/mimir/pkg/alertmanager/alertstore/local github.com/grafana/mimir/pkg/alertmanager/alertstore github.com/josharian/intern github.com/mailru/easyjson/jlexer github.com/grafana/mimir/pkg/ingester/client github.com/mailru/easyjson/buffer github.com/mailru/easyjson/jwriter github.com/go-openapi/swag github.com/prometheus/alertmanager/api/metrics github.com/hashicorp/golang-lru/v2/simplelru github.com/hashicorp/golang-lru/v2 github.com/prometheus/alertmanager/cluster github.com/cenkalti/backoff/v4 github.com/oklog/run github.com/prometheus/alertmanager/types github.com/grafana/mimir/pkg/storage/tsdb github.com/prometheus/alertmanager/provider github.com/prometheus/alertmanager/store github.com/prometheus/alertmanager/inhibit github.com/benbjohnson/clock github.com/prometheus/alertmanager/nflog/nflogpb github.com/grafana/mimir/cmd/metaconvert github.com/prometheus/alertmanager/nflog github.com/gofrs/uuid github.com/prometheus/alertmanager/silence/silencepb github.com/prometheus/alertmanager/silence github.com/shurcooL/httpfs/vfsutil github.com/shurcooL/vfsgen github.com/prometheus/alertmanager/asset golang.org/x/text/internal/tag golang.org/x/text/internal/language golang.org/x/text/internal/language/compact golang.org/x/text/language golang.org/x/text/internal golang.org/x/text/cases github.com/prometheus/alertmanager/template github.com/prometheus/alertmanager/notify github.com/prometheus/alertmanager/dispatch github.com/julienschmidt/httprouter github.com/prometheus/common/route github.com/prometheus/alertmanager/api/v1 github.com/go-openapi/analysis/internal/debug github.com/go-openapi/jsonpointer github.com/go-openapi/jsonreference/internal github.com/go-openapi/jsonreference github.com/go-openapi/spec github.com/go-openapi/analysis/internal/flatten/normalize github.com/go-openapi/analysis/internal/flatten/operations github.com/go-openapi/analysis/internal/flatten/replace github.com/go-openapi/analysis/internal/flatten/schutils github.com/go-openapi/analysis/internal/flatten/sortref github.com/asaskevich/govalidator github.com/go-openapi/errors go.mongodb.org/mongo-driver/bson/primitive go.mongodb.org/mongo-driver/x/bsonx/bsoncore go.mongodb.org/mongo-driver/bson/bsonrw go.mongodb.org/mongo-driver/bson/bsoncodec go.mongodb.org/mongo-driver/bson net/mail github.com/go-openapi/strfmt encoding/csv github.com/go-openapi/runtime/logger github.com/go-openapi/runtime/middleware/denco github.com/go-openapi/runtime/middleware/header github.com/docker/go-units github.com/go-openapi/runtime/flagext github.com/jessevdk/go-flags github.com/go-openapi/analysis github.com/go-openapi/loads github.com/go-openapi/runtime github.com/go-openapi/validate github.com/go-openapi/runtime/middleware/untyped github.com/go-openapi/runtime/security github.com/rs/cors github.com/grafana/mimir/pkg/util/net github.com/prometheus/alertmanager/notify/discord net/smtp github.com/prometheus/alertmanager/notify/email github.com/prometheus/alertmanager/notify/opsgenie github.com/go-openapi/runtime/middleware github.com/prometheus/alertmanager/api/v2/models github.com/prometheus/alertmanager/notify/pagerduty github.com/prometheus/alertmanager/api/v2/restapi/operations/alert github.com/prometheus/alertmanager/api/v2/restapi/operations/alertgroup github.com/prometheus/alertmanager/api/v2/restapi/operations/general github.com/prometheus/alertmanager/api/v2/restapi/operations/receiver github.com/prometheus/alertmanager/api/v2/restapi/operations/silence github.com/prometheus/alertmanager/notify/pushover github.com/prometheus/alertmanager/notify/slack github.com/aws/aws-sdk-go/service/sns github.com/prometheus/alertmanager/api/v2/restapi/operations github.com/prometheus/alertmanager/api/v2/restapi github.com/prometheus/alertmanager/api/v2 github.com/prometheus/alertmanager/notify/sns gopkg.in/telebot.v3 github.com/prometheus/alertmanager/api github.com/grafana/mimir/pkg/alertmanager/merger github.com/prometheus/alertmanager/notify/telegram github.com/prometheus/alertmanager/notify/victorops github.com/prometheus/alertmanager/notify/webex github.com/prometheus/alertmanager/notify/webhook github.com/prometheus/alertmanager/notify/wechat github.com/prometheus/alertmanager/provider/mem github.com/prometheus/alertmanager/ui github.com/google/pprof/profile github.com/grafana/mimir/pkg/alertmanager github.com/felixge/fgprof github.com/grafana/mimir/pkg/storage/tsdb/bucketindex github.com/grafana/mimir/pkg/ingester/activeseries github.com/grafana/mimir/pkg/util/extract github.com/grafana/mimir/pkg/util/globalerror github.com/grafana/dskit/limiter github.com/grafana/mimir/pkg/util/validation github.com/grafana/mimir/pkg/querier/stats github.com/grafana/mimir/pkg/util/httpgrpcutil go.opentelemetry.io/collector/featuregate go.opentelemetry.io/collector/pdata/internal/data go.opentelemetry.io/collector/pdata/internal/data/protogen/common/v1 github.com/grafana/mimir/pkg/compactor github.com/grafana/mimir/pkg/distributor/forwarding github.com/grafana/mimir/pkg/util/limiter go.opentelemetry.io/collector/pdata/internal/data/protogen/resource/v1 go.opentelemetry.io/collector/pdata/internal/data/protogen/logs/v1 go.opentelemetry.io/collector/pdata/internal/data/protogen/collector/logs/v1 go.opentelemetry.io/collector/pdata/internal/data/protogen/metrics/v1 go.opentelemetry.io/collector/pdata/internal/data/protogen/trace/v1 go.opentelemetry.io/collector/pdata/internal/data/protogen/collector/trace/v1 github.com/gogo/protobuf/jsonpb go.opentelemetry.io/collector/pdata/internal/data/protogen/collector/metrics/v1 go.opentelemetry.io/collector/pdata/internal go.opentelemetry.io/collector/pdata/pcommon go.opentelemetry.io/collector/pdata/internal/json go.opentelemetry.io/collector/pdata/internal/otlp go.opentelemetry.io/collector/pdata/pmetric/internal/pmetricjson github.com/prometheus/prometheus/prompb go.opentelemetry.io/collector/pdata/pmetric github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/prometheus go.opentelemetry.io/collector/pdata/pmetric/pmetricotlp github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/prometheusremotewrite github.com/prometheus/prometheus/util/osutil github.com/prometheus/prometheus/util/pool github.com/prometheus/prometheus/scrape github.com/grafana/mimir/pkg/util/push github.com/weaveworks/common/mtime github.com/grafana/mimir/pkg/distributor/distributorpb github.com/grafana/mimir/pkg/frontend/v1/frontendv1pb github.com/grafana/mimir/pkg/distributor github.com/grafana/mimir/pkg/scheduler/queue github.com/grafana/mimir/pkg/frontend/v1 github.com/grafana/mimir/pkg/frontend/v2/frontendv2pb github.com/grafana/mimir/pkg/util/servicediscovery github.com/grafana/mimir/pkg/scheduler/schedulerdiscovery github.com/grafana/mimir/pkg/scheduler/schedulerpb github.com/grafana/mimir/pkg/storage/chunk github.com/grafana/mimir/pkg/querier/batch github.com/grafana/mimir/pkg/frontend/v2 github.com/grafana/mimir/pkg/util/activitytracker github.com/grafana/mimir/pkg/querier/engine github.com/grafana/mimir/pkg/querier/iterators github.com/grafana/mimir/pkg/storage/lazyquery github.com/grafana/mimir/pkg/storage/series github.com/grafana/mimir/pkg/storegateway/chunkscache github.com/grafana/mimir/pkg/storegateway/storepb github.com/grafana/mimir/pkg/util/listblocks github.com/grafana/mimir/pkg/util/modelutil github.com/grafana/mimir/pkg/util/chunkcompat github.com/prometheus/prometheus/template github.com/prometheus/prometheus/model/rulefmt github.com/prometheus/prometheus/notifier github.com/grafana/mimir/pkg/storegateway/hintspb github.com/grafana/mimir/pkg/storegateway/storegatewaypb github.com/prometheus/prometheus/rules github.com/grafana/mimir/pkg/storegateway github.com/prometheus/prometheus/util/gate github.com/prometheus/prometheus/util/logging go.opentelemetry.io/otel/metric/instrument go.opentelemetry.io/otel/metric go.opentelemetry.io/otel/metric/internal/global go.opentelemetry.io/otel/metric/global go.opentelemetry.io/otel/semconv/v1.17.0 go.opentelemetry.io/otel/semconv/internal/v2 go.opentelemetry.io/otel/semconv/v1.17.0/httpconv go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp github.com/prometheus/prometheus/storage/remote github.com/grafana/mimir/pkg/ruler/rulespb github.com/grafana/mimir/pkg/ruler/rulestore/local github.com/grafana/mimir/pkg/ruler/rulestore github.com/grafana/mimir/pkg/ruler/rulestore/bucketclient github.com/grafana/mimir/pkg/util/version github.com/prometheus/client_golang/api github.com/prometheus/client_golang/api/prometheus/v1 github.com/grafana/mimir/pkg/querier github.com/prometheus/prometheus/discovery/refresh github.com/spf13/afero/mem github.com/spf13/afero golang.org/x/net/context/ctxhttp github.com/grafana/mimir/pkg/scheduler github.com/munnerz/goautoneg github.com/grafana/mimir/pkg/ruler github.com/prometheus/prometheus/util/httputil github.com/prometheus/prometheus/web/api/v1 github.com/grafana/mimir/pkg/usagestats github.com/grafana/mimir/pkg/util/gziphandler github.com/grafana/mimir/pkg/util/validation/exporter github.com/grafana/mimir/pkg/ingester github.com/grafana/mimir/pkg/api github.com/grafana/mimir/pkg/api/error github.com/grafana/mimir/pkg/frontend/querymiddleware/astmapper github.com/grafana/mimir/pkg/frontend/querymiddleware github.com/grafana/mimir/pkg/flusher github.com/grafana/mimir/pkg/frontend/transport github.com/grafana/mimir/pkg/util/instrumentation github.com/grafana/mimir/pkg/querier/tenantfederation github.com/grafana/mimir/pkg/querier/worker github.com/grafana/mimir/pkg/util/fs github.com/grafana/mimir/pkg/util/noauth github.com/grafana/mimir/pkg/util/process github.com/cenkalti/backoff/v3 github.com/hashicorp/go-retryablehttp github.com/grafana/mimir/pkg/frontend github.com/ryanuber/go-glob github.com/hashicorp/go-secure-stdlib/strutil github.com/hashicorp/go-secure-stdlib/parseutil github.com/hashicorp/hcl/hcl/strconv github.com/hashicorp/hcl/hcl/token golang.org/x/crypto/ed25519 github.com/hashicorp/hcl/hcl/ast github.com/hashicorp/hcl/hcl/scanner github.com/hashicorp/hcl/json/token github.com/hashicorp/hcl/hcl/parser github.com/hashicorp/hcl/json/scanner github.com/hashicorp/hcl/json/parser golang.org/x/crypto/pbkdf2 gopkg.in/square/go-jose.v2/cipher github.com/hashicorp/hcl gopkg.in/square/go-jose.v2/json github.com/grafana/mimir/pkg/util/fieldcategory github.com/grafana/mimir/pkg/util/usage github.com/grafana/mimir/pkg/continuoustest gopkg.in/square/go-jose.v2 database/sql gopkg.in/square/go-jose.v2/jwt github.com/prometheus/client_golang/prometheus/collectors github.com/grafana/mimir/cmd/mimir-continuous-test github.com/hashicorp/vault/api github.com/grafana/mimir/pkg/vault github.com/grafana/mimir/pkg/mimir github.com/grafana/mimir/cmd/mimir github.com/rainycape/unidecode github.com/gosimple/slug github.com/grafana-tools/sdk github.com/grafana/mimir/pkg/mimirtool/minisdk github.com/grafana/mimir/pkg/mimirtool/rules/rwrulefmt github.com/grafana/mimir/pkg/mimirtool/analyze github.com/grafana/mimir/pkg/mimirtool/backfill github.com/grafana/mimir/pkg/mimirtool/client github.com/grafana/mimir/tools/doc-generator/parse github.com/grafana/mimir/pkg/mimirtool/config github.com/dlclark/regexp2/syntax github.com/dlclark/regexp2 github.com/alecthomas/chroma github.com/alecthomas/chroma/formatters/html github.com/alecthomas/chroma/formatters/svg github.com/alecthomas/chroma/formatters github.com/alecthomas/chroma/lexers/internal github.com/alecthomas/chroma/lexers/a github.com/alecthomas/chroma/lexers/b github.com/alecthomas/chroma/lexers/p github.com/alecthomas/chroma/lexers/c github.com/alecthomas/chroma/lexers/j github.com/alecthomas/chroma/lexers/h github.com/alecthomas/chroma/lexers/circular github.com/alecthomas/chroma/lexers/d github.com/alecthomas/chroma/lexers/e github.com/alecthomas/chroma/lexers/f github.com/alecthomas/chroma/lexers/g github.com/alecthomas/chroma/lexers/i github.com/alecthomas/chroma/lexers/k github.com/alecthomas/chroma/lexers/l github.com/alecthomas/chroma/lexers/m github.com/alecthomas/chroma/lexers/n github.com/alecthomas/chroma/lexers/o github.com/alecthomas/chroma/lexers/q github.com/alecthomas/chroma/lexers/r github.com/alecthomas/chroma/lexers/t github.com/alecthomas/chroma/lexers/s github.com/alecthomas/chroma/lexers/v github.com/alecthomas/chroma/lexers/w github.com/alecthomas/chroma/lexers/x github.com/alecthomas/chroma/lexers/y github.com/alecthomas/chroma/lexers/z github.com/alecthomas/chroma/styles github.com/mitchellh/colorstring github.com/grafana/mimir/pkg/mimirtool/rules github.com/alecthomas/template/parse github.com/alecthomas/chroma/lexers github.com/alecthomas/chroma/quick github.com/grafana/mimir/pkg/mimirtool/printer go/scanner github.com/alecthomas/template go/ast go/doc/comment internal/lazyregexp github.com/google/go-querystring/query golang.org/x/crypto/openpgp/errors go/doc golang.org/x/crypto/openpgp/armor compress/bzip2 golang.org/x/crypto/cast5 golang.org/x/crypto/openpgp/elgamal gopkg.in/alecthomas/kingpin.v2 golang.org/x/crypto/openpgp/s2k image image/internal/imageutil image/jpeg golang.org/x/crypto/openpgp/packet github.com/grafana/mimir/pkg/mimirtool/commands golang.org/x/crypto/openpgp github.com/google/go-github/v32/github github.com/grafana/mimir/tools/querytee github.com/grafana/mimir/cmd/query-tee github.com/grafana/mimir/pkg/mimirtool/version github.com/grafana/mimir/cmd/mimirtool ==> Starting check()... === RUN TestConvertTenantBlocks --- PASS: TestConvertTenantBlocks (0.02s) === RUN TestConvertTenantBlocksDryMode --- PASS: TestConvertTenantBlocksDryMode (0.02s) PASS ok github.com/grafana/mimir/cmd/metaconvert 0.445s ? github.com/grafana/mimir/cmd/mimir-continuous-test [no test files] === RUN TestFlagParsing === RUN TestFlagParsing/common_yaml_unmarshaling_is_strict === RUN TestFlagParsing/help === RUN TestFlagParsing/user_visible_module_listing === RUN TestFlagParsing/user_visible_module_listing_flag_take_precedence_over_target_flag === RUN TestFlagParsing/common_yaml_inheritance_with_common_config_in_the_first_place === RUN TestFlagParsing/common_yaml_inheritance_with_common_config_in_the_last_place === RUN TestFlagParsing/common_yaml_sets_a_value_but_specific_config_reverts_it_back === RUN TestFlagParsing/default_values === RUN TestFlagParsing/common_yaml_overridden_by_a_common_flag_and_specific_flag === RUN TestFlagParsing/help-short === RUN TestFlagParsing/new_flag,_with_config === RUN TestFlagParsing/common_flag_inheritance === RUN TestFlagParsing/config === RUN TestFlagParsing/unknown_flag === RUN TestFlagParsing/root_level_configuration_option_specified_as_an_empty_node_in_YAML_does_not_set_entire_config_to_zero_value === RUN TestFlagParsing/version === RUN TestFlagParsing/help-all === RUN TestFlagParsing/config_with_arguments_override === RUN TestFlagParsing/config_with_expand-env --- PASS: TestFlagParsing (4.12s) --- PASS: TestFlagParsing/common_yaml_unmarshaling_is_strict (0.05s) --- PASS: TestFlagParsing/help (0.20s) --- PASS: TestFlagParsing/user_visible_module_listing (0.11s) --- PASS: TestFlagParsing/user_visible_module_listing_flag_take_precedence_over_target_flag (0.10s) --- PASS: TestFlagParsing/common_yaml_inheritance_with_common_config_in_the_first_place (0.31s) --- PASS: TestFlagParsing/common_yaml_inheritance_with_common_config_in_the_last_place (0.28s) --- PASS: TestFlagParsing/common_yaml_sets_a_value_but_specific_config_reverts_it_back (0.23s) --- PASS: TestFlagParsing/default_values (0.24s) --- PASS: TestFlagParsing/common_yaml_overridden_by_a_common_flag_and_specific_flag (0.31s) --- PASS: TestFlagParsing/help-short (0.18s) --- PASS: TestFlagParsing/new_flag,_with_config (0.19s) --- PASS: TestFlagParsing/common_flag_inheritance (0.29s) --- PASS: TestFlagParsing/config (0.18s) --- PASS: TestFlagParsing/unknown_flag (0.17s) --- PASS: TestFlagParsing/root_level_configuration_option_specified_as_an_empty_node_in_YAML_does_not_set_entire_config_to_zero_value (0.30s) --- PASS: TestFlagParsing/version (0.06s) --- PASS: TestFlagParsing/help-all (0.37s) --- PASS: TestFlagParsing/config_with_arguments_override (0.28s) --- PASS: TestFlagParsing/config_with_expand-env (0.29s) === RUN TestHelp === RUN TestHelp/basic === RUN TestHelp/all --- PASS: TestHelp (0.55s) --- PASS: TestHelp/basic (0.17s) --- PASS: TestHelp/all (0.37s) === RUN TestExpandEnvironmentVariables === RUN TestExpandEnvironmentVariables/x$y === RUN TestExpandEnvironmentVariables/x${y} === RUN TestExpandEnvironmentVariables/x$Y === RUN TestExpandEnvironmentVariables/x${Y} === RUN TestExpandEnvironmentVariables/x${Z:D} === RUN TestExpandEnvironmentVariables/x${Z:A_B_C_D} === RUN TestExpandEnvironmentVariables/x${Z:} === RUN TestExpandEnvironmentVariables/x$y:D --- PASS: TestExpandEnvironmentVariables (0.00s) --- PASS: TestExpandEnvironmentVariables/x$y (0.00s) --- PASS: TestExpandEnvironmentVariables/x${y} (0.00s) --- PASS: TestExpandEnvironmentVariables/x$Y (0.00s) --- PASS: TestExpandEnvironmentVariables/x${Y} (0.00s) --- PASS: TestExpandEnvironmentVariables/x${Z:D} (0.00s) --- PASS: TestExpandEnvironmentVariables/x${Z:A_B_C_D} (0.00s) --- PASS: TestExpandEnvironmentVariables/x${Z:} (0.00s) --- PASS: TestExpandEnvironmentVariables/x$y:D (0.00s) === RUN TestParseConfigFileParameter === RUN TestParseConfigFileParameter/#00 === RUN TestParseConfigFileParameter/--foo === RUN TestParseConfigFileParameter/-f_-a === RUN TestParseConfigFileParameter/--config.file=foo === RUN TestParseConfigFileParameter/--config.file_foo === RUN TestParseConfigFileParameter/--config.file=foo_--config.expand-env === RUN TestParseConfigFileParameter/--config.expand-env_--config.file=foo === RUN TestParseConfigFileParameter/--opt1_--config.file=foo === RUN TestParseConfigFileParameter/--opt1_--config.file_foo === RUN TestParseConfigFileParameter/--opt1_--config.file=foo_--config.expand-env === RUN TestParseConfigFileParameter/--opt1_--config.expand-env_--config.file=foo === RUN TestParseConfigFileParameter/--config.file=foo_--opt1 === RUN TestParseConfigFileParameter/--config.file_foo_--opt1 === RUN TestParseConfigFileParameter/--config.file=foo_--config.expand-env_--opt1 === RUN TestParseConfigFileParameter/--config.expand-env_--config.file=foo_--opt1 === RUN TestParseConfigFileParameter/--config.file=foo_--opt1_--config.expand-env === RUN TestParseConfigFileParameter/--config.expand-env_--opt1_--config.file=foo --- PASS: TestParseConfigFileParameter (0.00s) --- PASS: TestParseConfigFileParameter/#00 (0.00s) --- PASS: TestParseConfigFileParameter/--foo (0.00s) --- PASS: TestParseConfigFileParameter/-f_-a (0.00s) --- PASS: TestParseConfigFileParameter/--config.file=foo (0.00s) --- PASS: TestParseConfigFileParameter/--config.file_foo (0.00s) --- PASS: TestParseConfigFileParameter/--config.file=foo_--config.expand-env (0.00s) --- PASS: TestParseConfigFileParameter/--config.expand-env_--config.file=foo (0.00s) --- PASS: TestParseConfigFileParameter/--opt1_--config.file=foo (0.00s) --- PASS: TestParseConfigFileParameter/--opt1_--config.file_foo (0.00s) --- PASS: TestParseConfigFileParameter/--opt1_--config.file=foo_--config.expand-env (0.00s) --- PASS: TestParseConfigFileParameter/--opt1_--config.expand-env_--config.file=foo (0.00s) --- PASS: TestParseConfigFileParameter/--config.file=foo_--opt1 (0.00s) --- PASS: TestParseConfigFileParameter/--config.file_foo_--opt1 (0.00s) --- PASS: TestParseConfigFileParameter/--config.file=foo_--config.expand-env_--opt1 (0.00s) --- PASS: TestParseConfigFileParameter/--config.expand-env_--config.file=foo_--opt1 (0.00s) --- PASS: TestParseConfigFileParameter/--config.file=foo_--opt1_--config.expand-env (0.00s) --- PASS: TestParseConfigFileParameter/--config.expand-env_--opt1_--config.file=foo (0.00s) === RUN TestFieldCategoryOverridesNotStale --- PASS: TestFieldCategoryOverridesNotStale (0.10s) PASS ok github.com/grafana/mimir/cmd/mimir 5.448s ? github.com/grafana/mimir/cmd/mimirtool [no test files] ? github.com/grafana/mimir/integration [no test files] === RUN TestMimirReadRoutes --- PASS: TestMimirReadRoutes (0.00s) PASS ok github.com/grafana/mimir/cmd/query-tee 0.157s ? github.com/grafana/mimir/integration/ca [no test files] ? github.com/grafana/mimir/integration/e2emimir [no test files] === RUN TestResolveConfigs === RUN TestResolveConfigs/ResolveConfigs_annotates_values_that_match_defaults === RUN TestResolveConfigs/ResolveConfigs_sets_configuration_parameters_from_cli_arguments === RUN TestResolveConfigs/ResolveConfigs_sets_configuration_parameters_from_mounted_ConfigMaps === RUN TestResolveConfigs/ResolveConfigs_sets_configuration_parameters_from_mounted_Secrets --- PASS: TestResolveConfigs (2.12s) --- PASS: TestResolveConfigs/ResolveConfigs_annotates_values_that_match_defaults (0.59s) --- PASS: TestResolveConfigs/ResolveConfigs_sets_configuration_parameters_from_cli_arguments (0.48s) --- PASS: TestResolveConfigs/ResolveConfigs_sets_configuration_parameters_from_mounted_ConfigMaps (0.53s) --- PASS: TestResolveConfigs/ResolveConfigs_sets_configuration_parameters_from_mounted_Secrets (0.52s) PASS ok github.com/grafana/mimir/operations/compare-helm-with-jsonnet/plugins/resolve-config 2.576s ? github.com/grafana/mimir/pkg/alertmanager/alertmanagerpb [no test files] ? github.com/grafana/mimir/pkg/alertmanager/alertspb [no test files] === RUN TestMultitenantAlertmanager_GetStatusHandler --- PASS: TestMultitenantAlertmanager_GetStatusHandler (0.21s) === RUN TestAlertmanagerMetricsStore --- PASS: TestAlertmanagerMetricsStore (0.07s) === RUN TestAlertmanagerMetricsRemoval --- PASS: TestAlertmanagerMetricsRemoval (0.22s) === RUN TestIsHealthyForAlertmanagerOperations === PAUSE TestIsHealthyForAlertmanagerOperations === RUN Test_withCustomFunctions === RUN Test_withCustomFunctions/template_tenant_ID === RUN Test_withCustomFunctions/generate_grafana_explore_URL === RUN Test_withCustomFunctions/invalid_params_for_grafanaExploreURL --- PASS: Test_withCustomFunctions (0.03s) --- PASS: Test_withCustomFunctions/template_tenant_ID (0.00s) --- PASS: Test_withCustomFunctions/generate_grafana_explore_URL (0.00s) --- PASS: Test_withCustomFunctions/invalid_params_for_grafanaExploreURL (0.00s) === RUN TestDispatcherGroupLimits === RUN TestDispatcherGroupLimits/no_limit === RUN TestDispatcherGroupLimits/high_limit === RUN TestDispatcherGroupLimits/low_limit --- PASS: TestDispatcherGroupLimits (0.96s) --- PASS: TestDispatcherGroupLimits/no_limit (0.81s) --- PASS: TestDispatcherGroupLimits/high_limit (0.08s) --- PASS: TestDispatcherGroupLimits/low_limit (0.06s) === RUN TestDispatcherLoggerInsightKey --- PASS: TestDispatcherLoggerInsightKey (0.09s) === RUN TestAlertsLimiterWithNoLimits --- PASS: TestAlertsLimiterWithNoLimits (0.00s) === RUN TestAlertsLimiterWithCountLimit --- PASS: TestAlertsLimiterWithCountLimit (0.00s) === RUN TestAlertsLimiterWithSizeLimit --- PASS: TestAlertsLimiterWithSizeLimit (0.00s) === RUN TestAlertsLimiterWithSizeLimitAndAnnotationUpdate --- PASS: TestAlertsLimiterWithSizeLimitAndAnnotationUpdate (0.00s) === RUN TestAMConfigValidationAPI === RUN TestAMConfigValidationAPI/Should_return_error_if_the_alertmanager_config_contains_no_receivers === RUN TestAMConfigValidationAPI/Should_pass_if_the_alertmanager_config_is_valid === RUN TestAMConfigValidationAPI/Should_return_error_if_the_config_is_empty_due_to_wrong_indentation === RUN TestAMConfigValidationAPI/Should_return_error_if_the_alertmanager_config_is_empty_due_to_wrong_key === RUN TestAMConfigValidationAPI/Should_return_error_if_the_external_template_file_name_contains_an_absolute_path === RUN TestAMConfigValidationAPI/Should_return_error_if_the_external_template_file_name_contains_a_relative_path === RUN TestAMConfigValidationAPI/Should_return_error_if_the_external_template_file_name_is_not_a_valid_filename === RUN TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_the_root_/ === RUN TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_the_root_with_repeated_separators_/// === RUN TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_an_absolute_path === RUN TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_a_relative_path === RUN TestAMConfigValidationAPI/Should_pass_if_the_referenced_template_is_valid_filename === RUN TestAMConfigValidationAPI/Should_return_error_if_global_HTTP_password_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_HTTP_bearer_token_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_HTTP_credentials_file_is_set === RUN TestAMConfigValidationAPI/Should_NOT_return_error_if_global_HTTP_proxy_url_is_set === RUN TestAMConfigValidationAPI/Should_NOT_return_error_if_global_HTTP_proxy_from_environment_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_OAuth2_client_secret_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_OAuth2_proxy_url_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_OAuth2_proxy_from_environment_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_OAuth2_TLS_key_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_receiver's_HTTP_password_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_receiver's_HTTP_bearer_token_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_receiver's_HTTP_credentials_file_is_set === RUN TestAMConfigValidationAPI/Should_NOT_return_error_if_receiver's_HTTP_proxy_url_is_set === RUN TestAMConfigValidationAPI/Should_NOT_return_error_if_receiver's_HTTP_proxy_from_environment_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_receiver's_OAuth2_client_secret_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_receiver's_OAuth2_proxy_url_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_receiver's_OAuth2_proxy_from_environment_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_slack_api_url_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_Slack_api_url_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_opsgenie_api_key_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_OpsGenie_api_key_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_victorops_api_key_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_VictorOps_api_key_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_PagerDuty_service_key_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_PagerDuty_routing_key_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_Pushover_user_key_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_Pushover_token_file_is_set === RUN TestAMConfigValidationAPI/should_return_error_if_template_is_wrong === RUN TestAMConfigValidationAPI/config_too_big === RUN TestAMConfigValidationAPI/config_size_OK === RUN TestAMConfigValidationAPI/templates_limit_reached === RUN TestAMConfigValidationAPI/templates_limit_not_reached === RUN TestAMConfigValidationAPI/template_size_limit_reached === RUN TestAMConfigValidationAPI/template_size_limit_ok === RUN TestAMConfigValidationAPI/Should_pass_if_template_uses_the_tenantID_custom_function === RUN TestAMConfigValidationAPI/Should_pass_if_template_uses_the_grafanaExploreURL_custom_function --- PASS: TestAMConfigValidationAPI (0.57s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_alertmanager_config_contains_no_receivers (0.00s) --- PASS: TestAMConfigValidationAPI/Should_pass_if_the_alertmanager_config_is_valid (0.03s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_config_is_empty_due_to_wrong_indentation (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_alertmanager_config_is_empty_due_to_wrong_key (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_external_template_file_name_contains_an_absolute_path (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_external_template_file_name_contains_a_relative_path (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_external_template_file_name_is_not_a_valid_filename (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_the_root_/ (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_the_root_with_repeated_separators_/// (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_an_absolute_path (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_a_relative_path (0.00s) --- PASS: TestAMConfigValidationAPI/Should_pass_if_the_referenced_template_is_valid_filename (0.09s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_HTTP_password_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_HTTP_bearer_token_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_HTTP_credentials_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_NOT_return_error_if_global_HTTP_proxy_url_is_set (0.03s) --- PASS: TestAMConfigValidationAPI/Should_NOT_return_error_if_global_HTTP_proxy_from_environment_is_set (0.03s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_OAuth2_client_secret_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_OAuth2_proxy_url_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_OAuth2_proxy_from_environment_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_OAuth2_TLS_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_receiver's_HTTP_password_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_receiver's_HTTP_bearer_token_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_receiver's_HTTP_credentials_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_NOT_return_error_if_receiver's_HTTP_proxy_url_is_set (0.03s) --- PASS: TestAMConfigValidationAPI/Should_NOT_return_error_if_receiver's_HTTP_proxy_from_environment_is_set (0.04s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_receiver's_OAuth2_client_secret_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_receiver's_OAuth2_proxy_url_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_receiver's_OAuth2_proxy_from_environment_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_slack_api_url_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_Slack_api_url_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_opsgenie_api_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_OpsGenie_api_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_victorops_api_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_VictorOps_api_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_PagerDuty_service_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_PagerDuty_routing_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_Pushover_user_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_Pushover_token_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/should_return_error_if_template_is_wrong (0.03s) --- PASS: TestAMConfigValidationAPI/config_too_big (0.00s) --- PASS: TestAMConfigValidationAPI/config_size_OK (0.03s) --- PASS: TestAMConfigValidationAPI/templates_limit_reached (0.00s) --- PASS: TestAMConfigValidationAPI/templates_limit_not_reached (0.03s) --- PASS: TestAMConfigValidationAPI/template_size_limit_reached (0.00s) --- PASS: TestAMConfigValidationAPI/template_size_limit_ok (0.03s) --- PASS: TestAMConfigValidationAPI/Should_pass_if_template_uses_the_tenantID_custom_function (0.03s) --- PASS: TestAMConfigValidationAPI/Should_pass_if_template_uses_the_grafanaExploreURL_custom_function (0.03s) === RUN TestMultitenantAlertmanager_DeleteUserConfig --- PASS: TestMultitenantAlertmanager_DeleteUserConfig (0.00s) === RUN TestAMConfigListUserConfig --- PASS: TestAMConfigListUserConfig (0.23s) === RUN TestValidateAlertmanagerConfig === RUN TestValidateAlertmanagerConfig/map_containing_*HTTPClientConfig === RUN TestValidateAlertmanagerConfig/*HTTPClientConfig === RUN TestValidateAlertmanagerConfig/TLSConfig === RUN TestValidateAlertmanagerConfig/GlobalConfig.SMTPAuthPasswordFile === RUN TestValidateAlertmanagerConfig/*EmailConfig.AuthPasswordFile === RUN TestValidateAlertmanagerConfig/EmailConfig.AuthPasswordFile === RUN TestValidateAlertmanagerConfig/struct_containing_*HTTPClientConfig_as_nested_child_within_a_slice === RUN TestValidateAlertmanagerConfig/HTTPClientConfig === RUN TestValidateAlertmanagerConfig/*TLSConfig === RUN TestValidateAlertmanagerConfig/*GlobalConfig.SMTPAuthPasswordFile === RUN TestValidateAlertmanagerConfig/struct_containing_*HTTPClientConfig_as_direct_child === RUN TestValidateAlertmanagerConfig/struct_containing_*HTTPClientConfig_as_nested_child === RUN TestValidateAlertmanagerConfig/map_containing_TLSConfig_as_nested_child --- PASS: TestValidateAlertmanagerConfig (0.01s) --- PASS: TestValidateAlertmanagerConfig/map_containing_*HTTPClientConfig (0.00s) --- PASS: TestValidateAlertmanagerConfig/*HTTPClientConfig (0.00s) --- PASS: TestValidateAlertmanagerConfig/TLSConfig (0.00s) --- PASS: TestValidateAlertmanagerConfig/GlobalConfig.SMTPAuthPasswordFile (0.00s) --- PASS: TestValidateAlertmanagerConfig/*EmailConfig.AuthPasswordFile (0.00s) --- PASS: TestValidateAlertmanagerConfig/EmailConfig.AuthPasswordFile (0.00s) --- PASS: TestValidateAlertmanagerConfig/struct_containing_*HTTPClientConfig_as_nested_child_within_a_slice (0.00s) --- PASS: TestValidateAlertmanagerConfig/HTTPClientConfig (0.00s) --- PASS: TestValidateAlertmanagerConfig/*TLSConfig (0.00s) --- PASS: TestValidateAlertmanagerConfig/*GlobalConfig.SMTPAuthPasswordFile (0.00s) --- PASS: TestValidateAlertmanagerConfig/struct_containing_*HTTPClientConfig_as_direct_child (0.00s) --- PASS: TestValidateAlertmanagerConfig/struct_containing_*HTTPClientConfig_as_nested_child (0.00s) --- PASS: TestValidateAlertmanagerConfig/map_containing_TLSConfig_as_nested_child (0.00s) === RUN TestDistributor_DistributeRequest === RUN TestDistributor_DistributeRequest/Write_/alerts,_Simple_AM_request,_all_AM_healthy === RUN TestDistributor_DistributeRequest/Write_/alerts,_no_healthy_Alertmanagers === RUN TestDistributor_DistributeRequest/Write_/alerts,_1_healthy_Alertmanager_out_of_3 === RUN TestDistributor_DistributeRequest/Write_/alerts,_3_healthy_Alertmanagers_out_of_5 === RUN TestDistributor_DistributeRequest/Read_/v1/alerts_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Read_/v2/alerts_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Read_/v2/alerts/groups_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Write_/alerts/groups_not_supported === RUN TestDistributor_DistributeRequest/Read_/v1/silences_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Read_/v2/silences_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Write_/silences_is_sent_to_only_1_AM === RUN TestDistributor_DistributeRequest/Read_/v1/silence/id_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Read_/v2/silence/id_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Write_/silence/id_not_supported === RUN TestDistributor_DistributeRequest/Delete_/silence/id_is_sent_to_only_1_AM === RUN TestDistributor_DistributeRequest/Read_/status_is_sent_to_only_1_AM === RUN TestDistributor_DistributeRequest/Write_/status_not_supported === RUN TestDistributor_DistributeRequest/Read_/some/place/arbitrary_is_sent_to_only_1_AM === RUN TestDistributor_DistributeRequest/Read_/receivers_is_sent_to_only_1_AM === RUN TestDistributor_DistributeRequest/Write_/receivers_not_supported --- PASS: TestDistributor_DistributeRequest (0.21s) --- PASS: TestDistributor_DistributeRequest/Write_/alerts,_Simple_AM_request,_all_AM_healthy (0.02s) --- PASS: TestDistributor_DistributeRequest/Write_/alerts,_no_healthy_Alertmanagers (0.01s) --- PASS: TestDistributor_DistributeRequest/Write_/alerts,_1_healthy_Alertmanager_out_of_3 (0.01s) --- PASS: TestDistributor_DistributeRequest/Write_/alerts,_3_healthy_Alertmanagers_out_of_5 (0.01s) --- PASS: TestDistributor_DistributeRequest/Read_/v1/alerts_is_sent_to_3_AMs (0.02s) --- PASS: TestDistributor_DistributeRequest/Read_/v2/alerts_is_sent_to_3_AMs (0.02s) --- PASS: TestDistributor_DistributeRequest/Read_/v2/alerts/groups_is_sent_to_3_AMs (0.02s) --- PASS: TestDistributor_DistributeRequest/Write_/alerts/groups_not_supported (0.00s) --- PASS: TestDistributor_DistributeRequest/Read_/v1/silences_is_sent_to_3_AMs (0.02s) --- PASS: TestDistributor_DistributeRequest/Read_/v2/silences_is_sent_to_3_AMs (0.02s) --- PASS: TestDistributor_DistributeRequest/Write_/silences_is_sent_to_only_1_AM (0.00s) --- PASS: TestDistributor_DistributeRequest/Read_/v1/silence/id_is_sent_to_3_AMs (0.03s) --- PASS: TestDistributor_DistributeRequest/Read_/v2/silence/id_is_sent_to_3_AMs (0.02s) --- PASS: TestDistributor_DistributeRequest/Write_/silence/id_not_supported (0.00s) --- PASS: TestDistributor_DistributeRequest/Delete_/silence/id_is_sent_to_only_1_AM (0.00s) --- PASS: TestDistributor_DistributeRequest/Read_/status_is_sent_to_only_1_AM (0.00s) --- PASS: TestDistributor_DistributeRequest/Write_/status_not_supported (0.00s) --- PASS: TestDistributor_DistributeRequest/Read_/some/place/arbitrary_is_sent_to_only_1_AM (0.00s) --- PASS: TestDistributor_DistributeRequest/Read_/receivers_is_sent_to_only_1_AM (0.00s) --- PASS: TestDistributor_DistributeRequest/Write_/receivers_not_supported (0.00s) === RUN TestMultitenantAlertmanagerConfig_Validate === RUN TestMultitenantAlertmanagerConfig_Validate/should_pass_if_the_URL_just_contains_the_path_with_the_leading_/ === RUN TestMultitenantAlertmanagerConfig_Validate/should_pass_with_default_config === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_with_empty_external_URL === RUN TestMultitenantAlertmanagerConfig_Validate/should_succeed_if_external_URL_does_not_end_with_/ === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_external_URL_has_no_hostname === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_zone_aware_is_enabled_but_zone_is_not_set === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_the_URL_just_contains_the_hostname_(because_it_can't_be_disambiguated_with_a_path) === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_persistent_interval_is_0 === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_persistent_interval_is_negative === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_external_URL_ends_with_/ === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_external_URL_has_no_scheme --- PASS: TestMultitenantAlertmanagerConfig_Validate (0.03s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_pass_if_the_URL_just_contains_the_path_with_the_leading_/ (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_pass_with_default_config (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_with_empty_external_URL (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_succeed_if_external_URL_does_not_end_with_/ (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_external_URL_has_no_hostname (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_zone_aware_is_enabled_but_zone_is_not_set (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_the_URL_just_contains_the_hostname_(because_it_can't_be_disambiguated_with_a_path) (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_persistent_interval_is_0 (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_persistent_interval_is_negative (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_external_URL_ends_with_/ (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_external_URL_has_no_scheme (0.00s) === RUN TestMultitenantAlertmanager_relativeDataDir --- PASS: TestMultitenantAlertmanager_relativeDataDir (0.36s) === RUN TestMultitenantAlertmanager_loadAndSyncConfigs --- PASS: TestMultitenantAlertmanager_loadAndSyncConfigs (0.46s) === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webex_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webex_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webex_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webex_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=false === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=true === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=true === NAME TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=true multitenant_test.go:693: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=debug component=MultiTenantAlertmanager user=user-1 component=nflog msg="notification log snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=HTTP_proxy_firewall_enabled=true2933987189/001/user-1/notifications: no such file or directory" level=debug component=MultiTenantAlertmanager user=user-1 component=silences msg="silences snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=HTTP_proxy_firewall_enabled=true2933987189/001/user-1/silences: no such file or directory" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=webhook integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://www.google.com\": proxyconnect tcp: dial tcp 127.0.0.1:33575: blocked address" === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=false === NAME TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=true multitenant_test.go:693: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=debug component=MultiTenantAlertmanager user=user-1 component=nflog msg="notification log snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=telegram_firewall_enabled=true1444917111/001/user-1/notifications: no such file or directory" level=debug component=MultiTenantAlertmanager user=user-1 component=silences msg="silences snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=telegram_firewall_enabled=true1444917111/001/user-1/silences: no such file or directory" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=telegram integration=telegram[0] msg="Notify attempt failed, will retry later" attempts=1 err="telebot: Post \"http://127.0.0.1:34347/botxxx/sendMessage\": dial tcp 127.0.0.1:34347: blocked address" === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=true === NAME TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=false multitenant_test.go:693: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=debug component=MultiTenantAlertmanager user=user-1 component=nflog msg="notification log snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=sns_firewall_enabled=false2817095346/001/user-1/notifications: no such file or directory" level=debug component=MultiTenantAlertmanager user=user-1 component=silences msg="silences snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=sns_firewall_enabled=false2817095346/001/user-1/silences: no such file or directory" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=sns msg="SNS message successfully published" message_id=null sequencenumber=null level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=sns integration=sns[0] msg="Notify success" attempts=1 === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=false multitenant_test.go:693: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="ring doesn't exist in KV store yet" level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=debug component=MultiTenantAlertmanager user=user-1 component=nflog msg="notification log snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=wechat_firewall_enabled=false2415866251/001/user-1/notifications: no such file or directory" level=debug component=MultiTenantAlertmanager user=user-1 component=silences msg="silences snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=wechat_firewall_enabled=false2415866251/001/user-1/silences: no such file or directory" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=wechat incident={}:{} level=error component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="wechat/wechat[0]: notify retry canceled due to unrecoverable error after 1 attempts: EOF" === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=true === NAME TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=true multitenant_test.go:693: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=debug component=MultiTenantAlertmanager user=user-1 component=nflog msg="notification log snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=sns_firewall_enabled=true2622198816/001/user-1/notifications: no such file or directory" level=debug component=MultiTenantAlertmanager user=user-1 component=silences msg="silences snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=sns_firewall_enabled=true2622198816/001/user-1/silences: no such file or directory" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=sns integration=sns[0] msg="Notify attempt failed, will retry later" attempts=1 err="RequestError: send request failed\ncaused by: Post \"http://127.0.0.1:36763/\": dial tcp 127.0.0.1:36763: blocked address" === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=false multitenant_test.go:693: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="ring doesn't exist in KV store yet" level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=debug component=MultiTenantAlertmanager user=user-1 component=nflog msg="notification log snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=opsgenie_firewall_enabled=false1033713105/001/user-1/notifications: no such file or directory" level=debug component=MultiTenantAlertmanager user=user-1 component=silences msg="silences snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=opsgenie_firewall_enabled=false1033713105/001/user-1/silences: no such file or directory" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=opsgenie alert={}:{} level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=opsgenie integration=opsgenie[0] msg="Notify success" attempts=1 === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=true === NAME TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=true multitenant_test.go:693: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=debug component=MultiTenantAlertmanager user=user-1 component=nflog msg="notification log snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=wechat_firewall_enabled=true4102106771/001/user-1/notifications: no such file or directory" level=debug component=MultiTenantAlertmanager user=user-1 component=silences msg="silences snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=wechat_firewall_enabled=true4102106771/001/user-1/silences: no such file or directory" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=wechat incident={}:{} level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=wechat integration=wechat[0] msg="Notify attempt failed, will retry later" attempts=1 err="Get \"\": dial tcp 127.0.0.1:35957: blocked address" level=debug component=MultiTenantAlertmanager user=user-1 integration=wechat incident={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=wechat incident={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=wechat incident={}:{} === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=false multitenant_test.go:693: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=debug component=MultiTenantAlertmanager user=user-1 component=nflog msg="notification log snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=webhook_firewall_enabled=false3182503990/001/user-1/notifications: no such file or directory" level=debug component=MultiTenantAlertmanager user=user-1 component=silences msg="silences snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=webhook_firewall_enabled=false3182503990/001/user-1/silences: no such file or directory" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=webhook integration=webhook[0] msg="Notify success" attempts=1 === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=true === NAME TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=true multitenant_test.go:693: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="ring doesn't exist in KV store yet" level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=debug component=MultiTenantAlertmanager user=user-1 component=nflog msg="notification log snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=opsgenie_firewall_enabled=true4209920765/001/user-1/notifications: no such file or directory" level=debug component=MultiTenantAlertmanager user=user-1 component=silences msg="silences snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=opsgenie_firewall_enabled=true4209920765/001/user-1/silences: no such file or directory" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=opsgenie alert={}:{} level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=opsgenie integration=opsgenie[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"http://127.0.0.1:40831/v2/alerts\": dial tcp 127.0.0.1:40831: blocked address" level=debug component=MultiTenantAlertmanager user=user-1 integration=opsgenie alert={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=opsgenie alert={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=opsgenie alert={}:{} === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=false multitenant_test.go:693: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=debug component=MultiTenantAlertmanager user=user-1 component=nflog msg="notification log snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=telegram_firewall_enabled=false1993825554/001/user-1/notifications: no such file or directory" level=debug component=MultiTenantAlertmanager user=user-1 component=silences msg="silences snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=telegram_firewall_enabled=false1993825554/001/user-1/silences: no such file or directory" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=telegram integration=telegram[0] msg="Notify attempt failed, will retry later" attempts=1 err="telebot: unexpected end of JSON input" === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=false === NAME TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=true multitenant_test.go:693: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=debug component=MultiTenantAlertmanager user=user-1 component=nflog msg="notification log snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=webhook_firewall_enabled=true2339020204/001/user-1/notifications: no such file or directory" level=debug component=MultiTenantAlertmanager user=user-1 component=silences msg="silences snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=webhook_firewall_enabled=true2339020204/001/user-1/silences: no such file or directory" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=webhook integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"http://127.0.0.1:36089\": dial tcp 127.0.0.1:36089: blocked address" === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webex_firewall_enabled=false === NAME TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=false multitenant_test.go:693: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=debug component=MultiTenantAlertmanager user=user-1 component=nflog msg="notification log snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=slack_firewall_enabled=false2495280988/001/user-1/notifications: no such file or directory" level=debug component=MultiTenantAlertmanager user=user-1 component=silences msg="silences snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=slack_firewall_enabled=false2495280988/001/user-1/silences: no such file or directory" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=slack integration=slack[0] msg="Notify success" attempts=1 === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webex_firewall_enabled=true === NAME TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webex_firewall_enabled=false multitenant_test.go:693: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=debug component=MultiTenantAlertmanager user=user-1 component=nflog msg="notification log snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=webex_firewall_enabled=false2504912355/001/user-1/notifications: no such file or directory" level=debug component=MultiTenantAlertmanager user=user-1 component=silences msg="silences snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=webex_firewall_enabled=false2504912355/001/user-1/silences: no such file or directory" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=webex incident={}:{} level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=webex integration=webex[0] msg="Notify success" attempts=1 === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=false multitenant_test.go:693: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="ring doesn't exist in KV store yet" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=debug component=MultiTenantAlertmanager user=user-1 component=nflog msg="notification log snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=discord_firewall_enabled=false3441052191/001/user-1/notifications: no such file or directory" level=debug component=MultiTenantAlertmanager user=user-1 component=silences msg="silences snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=discord_firewall_enabled=false3441052191/001/user-1/silences: no such file or directory" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=discord incident={}:{} level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=discord integration=discord[0] msg="Notify success" attempts=1 === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=true === NAME TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webex_firewall_enabled=true multitenant_test.go:693: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=debug component=MultiTenantAlertmanager user=user-1 component=nflog msg="notification log snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=webex_firewall_enabled=true2228166212/001/user-1/notifications: no such file or directory" level=debug component=MultiTenantAlertmanager user=user-1 component=silences msg="silences snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=webex_firewall_enabled=true2228166212/001/user-1/silences: no such file or directory" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=webex incident={}:{} level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=webex integration=webex[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"\": dial tcp 127.0.0.1:39457: blocked address" level=debug component=MultiTenantAlertmanager user=user-1 integration=webex incident={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=webex incident={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=webex incident={}:{} === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=false === NAME TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=true multitenant_test.go:693: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=debug component=MultiTenantAlertmanager user=user-1 component=nflog msg="notification log snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=discord_firewall_enabled=true373491918/001/user-1/notifications: no such file or directory" level=debug component=MultiTenantAlertmanager user=user-1 component=silences msg="silences snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=discord_firewall_enabled=true373491918/001/user-1/silences: no such file or directory" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=discord incident={}:{} level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=discord integration=discord[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"\": dial tcp 127.0.0.1:35285: blocked address" level=debug component=MultiTenantAlertmanager user=user-1 integration=discord incident={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=discord incident={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=discord incident={}:{} === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=true === NAME TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=false multitenant_test.go:693: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=debug component=MultiTenantAlertmanager user=user-1 component=nflog msg="notification log snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=pagerduty_firewall_enabled=false1232133264/001/user-1/notifications: no such file or directory" level=debug component=MultiTenantAlertmanager user=user-1 component=silences msg="silences snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=pagerduty_firewall_enabled=false1232133264/001/user-1/silences: no such file or directory" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=pagerduty incident={}:{} eventType=trigger level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=pagerduty integration=pagerduty[0] msg="Notify success" attempts=1 === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=true multitenant_test.go:693: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="ring doesn't exist in KV store yet" level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=debug component=MultiTenantAlertmanager user=user-1 component=nflog msg="notification log snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=pagerduty_firewall_enabled=true1798600245/001/user-1/notifications: no such file or directory" level=debug component=MultiTenantAlertmanager user=user-1 component=silences msg="silences snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=pagerduty_firewall_enabled=true1798600245/001/user-1/silences: no such file or directory" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=pagerduty incident={}:{} eventType=trigger level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=pagerduty integration=pagerduty[0] msg="Notify attempt failed, will retry later" attempts=1 err="failed to post message to PagerDuty: Post \"http://127.0.0.1:45199\": dial tcp 127.0.0.1:45199: blocked address" level=debug component=MultiTenantAlertmanager user=user-1 integration=pagerduty incident={}:{} eventType=trigger level=debug component=MultiTenantAlertmanager user=user-1 integration=pagerduty incident={}:{} eventType=trigger level=debug component=MultiTenantAlertmanager user=user-1 integration=pagerduty incident={}:{} eventType=trigger === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=false === NAME TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=true multitenant_test.go:693: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="ring doesn't exist in KV store yet" level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=debug component=MultiTenantAlertmanager user=user-1 component=nflog msg="notification log snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=slack_firewall_enabled=true4189708134/001/user-1/notifications: no such file or directory" level=debug component=MultiTenantAlertmanager user=user-1 component=silences msg="silences snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=slack_firewall_enabled=true4189708134/001/user-1/silences: no such file or directory" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=slack integration=slack[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"\": dial tcp 127.0.0.1:42945: blocked address" === NAME TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=false multitenant_test.go:693: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=debug component=MultiTenantAlertmanager user=user-1 component=nflog msg="notification log snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=HTTP_proxy_firewall_enabled=false3642258990/001/user-1/notifications: no such file or directory" level=debug component=MultiTenantAlertmanager user=user-1 component=silences msg="silences snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabledreceiver=HTTP_proxy_firewall_enabled=false3642258990/001/user-1/silences: no such file or directory" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=webhook integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://www.google.com\": http: server gave HTTP response to HTTPS client" --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled (0.00s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=true (3.32s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=true (3.41s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=false (0.53s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=false (0.47s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=true (3.27s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=false (0.62s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=true (3.28s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=false (0.11s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=true (3.38s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=false (0.20s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=true (3.30s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=false (0.53s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webex_firewall_enabled=false (0.51s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=false (0.20s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webex_firewall_enabled=true (3.12s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=true (3.19s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=false (0.47s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=true (3.41s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=true (3.55s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=false (0.54s) === RUN TestMultitenantAlertmanager_deleteUnusedLocalUserState --- PASS: TestMultitenantAlertmanager_deleteUnusedLocalUserState (0.26s) === RUN TestMultitenantAlertmanager_zoneAwareSharding level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=instance-1 ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=instance-2 ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=instance-3 ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=2 level=debug component=MultiTenantAlertmanager msg="setting config" user=user1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user1 level=debug component=MultiTenantAlertmanager user=user1 component=nflog msg="notification log snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_zoneAwareSharding1618289809/001/user1/notifications: no such file or directory" level=debug component=MultiTenantAlertmanager user=user1 component=silences msg="silences snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_zoneAwareSharding1618289809/001/user1/silences: no such file or directory" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="Waiting for notification and silences to settle..." level=debug component=MultiTenantAlertmanager msg="contacting replica for full state" user=user1 addr=127.0.0.1-3:0 level=debug component=MultiTenantAlertmanager msg="setting config" user=user2 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user2 level=debug component=MultiTenantAlertmanager user=user2 component=nflog msg="notification log snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_zoneAwareSharding1618289809/001/user2/notifications: no such file or directory" level=debug component=MultiTenantAlertmanager user=user2 component=silences msg="silences snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_zoneAwareSharding1618289809/001/user2/silences: no such file or directory" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="Waiting for notification and silences to settle..." level=debug component=MultiTenantAlertmanager msg="contacting replica for full state" user=user2 addr=127.0.0.1-3:0 level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user3 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user3 level=debug component=MultiTenantAlertmanager user=user3 component=nflog msg="notification log snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_zoneAwareSharding1618289809/002/user3/notifications: no such file or directory" level=debug component=MultiTenantAlertmanager user=user3 component=silences msg="silences snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_zoneAwareSharding1618289809/002/user3/silences: no such file or directory" level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="Waiting for notification and silences to settle..." level=debug component=MultiTenantAlertmanager msg="contacting replica for full state" user=user3 addr=127.0.0.1-3:0 level=error component=MultiTenantAlertmanager msg="rpc reading state from replica failed" addr=127.0.0.1-3:0 user=user3 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup 127.0.0.1-3: no such host\"" level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="unable to read state from other Alertmanager replicas; trying to read from storage" err="failed to read state from any replica" level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="no state for user in storage; proceeding" level=error component=MultiTenantAlertmanager msg="rpc reading state from replica failed" addr=127.0.0.1-3:0 user=user1 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup 127.0.0.1-3: no such host\"" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="unable to read state from other Alertmanager replicas; trying to read from storage" err="failed to read state from any replica" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="no state for user in storage; proceeding" level=error component=MultiTenantAlertmanager msg="rpc reading state from replica failed" addr=127.0.0.1-3:0 user=user2 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup 127.0.0.1-3: no such host\"" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="unable to read state from other Alertmanager replicas; trying to read from storage" err="failed to read state from any replica" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=3 level=debug component=MultiTenantAlertmanager msg="setting config" user=user2 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user2 level=debug component=MultiTenantAlertmanager user=user2 component=nflog msg="notification log snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_zoneAwareSharding1618289809/003/user2/notifications: no such file or directory" level=debug component=MultiTenantAlertmanager user=user2 component=silences msg="silences snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_zoneAwareSharding1618289809/003/user2/silences: no such file or directory" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="Waiting for notification and silences to settle..." level=debug component=MultiTenantAlertmanager msg="contacting replica for full state" user=user2 addr=127.0.0.1-1:0 level=debug component=MultiTenantAlertmanager msg="setting config" user=user3 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user3 level=debug component=MultiTenantAlertmanager user=user3 component=nflog msg="notification log snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_zoneAwareSharding1618289809/003/user3/notifications: no such file or directory" level=debug component=MultiTenantAlertmanager user=user3 component=silences msg="silences snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_zoneAwareSharding1618289809/003/user3/silences: no such file or directory" level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="Waiting for notification and silences to settle..." level=debug component=MultiTenantAlertmanager msg="contacting replica for full state" user=user3 addr=127.0.0.1-2:0 level=debug component=MultiTenantAlertmanager msg="setting config" user=user1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user1 level=debug component=MultiTenantAlertmanager user=user1 component=nflog msg="notification log snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_zoneAwareSharding1618289809/003/user1/notifications: no such file or directory" level=debug component=MultiTenantAlertmanager user=user1 component=silences msg="silences snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_zoneAwareSharding1618289809/003/user1/silences: no such file or directory" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="Waiting for notification and silences to settle..." level=debug component=MultiTenantAlertmanager msg="contacting replica for full state" user=user1 addr=127.0.0.1-1:0 level=error component=MultiTenantAlertmanager msg="rpc reading state from replica failed" addr=127.0.0.1-1:0 user=user2 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup 127.0.0.1-1: no such host\"" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="unable to read state from other Alertmanager replicas; trying to read from storage" err="failed to read state from any replica" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="no state for user in storage; proceeding" level=error component=MultiTenantAlertmanager msg="rpc reading state from replica failed" addr=127.0.0.1-1:0 user=user1 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup 127.0.0.1-1: no such host\"" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="unable to read state from other Alertmanager replicas; trying to read from storage" err="failed to read state from any replica" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="no state for user in storage; proceeding" level=debug component=MultiTenantAlertmanager user=user2 component=nflog msg="Running maintenance" level=info component=MultiTenantAlertmanager msg="ring lifecycler is shutting down" ring=alertmanager level=info component=MultiTenantAlertmanager msg="unregistering instance from ring" ring=alertmanager level=debug component=MultiTenantAlertmanager user=user2 component=nflog msg="Maintenance done" duration=3.138092ms size=0 level=debug component=MultiTenantAlertmanager user=user2 component=silences msg="Running maintenance" level=info component=MultiTenantAlertmanager msg="instance removed from the ring" ring=alertmanager level=debug component=MultiTenantAlertmanager user=user2 component=silences msg="Maintenance done" duration=1.757057ms size=0 level=debug component=MultiTenantAlertmanager user=user3 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user3 component=nflog msg="Maintenance done" duration=479.866µs size=0 level=warn component=MultiTenantAlertmanager user=user3 msg="error while stopping state persister service" err="invalid service state: Failed, expected: Terminated, failure: context canceled" level=debug component=MultiTenantAlertmanager user=user3 component=silences msg="Running maintenance" level=error component=MultiTenantAlertmanager msg="rpc reading state from replica failed" addr=127.0.0.1-2:0 user=user3 err="rpc error: code = Canceled desc = context canceled" level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="unable to read state from other Alertmanager replicas; trying to read from storage" err="context canceled" level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="no state for user in storage; proceeding" level=debug component=MultiTenantAlertmanager user=user3 component=silences msg="Maintenance done" duration=4.02294ms size=0 level=debug component=MultiTenantAlertmanager user=user1 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user1 component=silences msg="Maintenance done" duration=435.064µs size=0 level=debug component=MultiTenantAlertmanager user=user1 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user1 component=nflog msg="Maintenance done" duration=1.215746ms size=0 level=debug component=MultiTenantAlertmanager user=user3 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user3 component=nflog msg="Maintenance done" duration=484.666µs size=0 level=debug component=MultiTenantAlertmanager user=user3 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user3 component=silences msg="Maintenance done" duration=419.383µs size=0 level=info component=MultiTenantAlertmanager msg="ring lifecycler is shutting down" ring=alertmanager level=info component=MultiTenantAlertmanager msg="unregistering instance from ring" ring=alertmanager level=info component=MultiTenantAlertmanager msg="instance removed from the ring" ring=alertmanager level=debug component=MultiTenantAlertmanager user=user1 component=nflog msg="Running maintenance" level=info component=MultiTenantAlertmanager msg="ring lifecycler is shutting down" ring=alertmanager level=info component=MultiTenantAlertmanager msg="unregistering instance from ring" ring=alertmanager level=info component=MultiTenantAlertmanager msg="instance removed from the ring" ring=alertmanager level=debug component=MultiTenantAlertmanager user=user1 component=nflog msg="Maintenance done" duration=5.66079ms size=0 level=debug component=MultiTenantAlertmanager user=user1 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user1 component=silences msg="Maintenance done" duration=767.562µs size=0 level=debug component=MultiTenantAlertmanager user=user2 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user2 component=nflog msg="Maintenance done" duration=629.795µs size=0 level=debug component=MultiTenantAlertmanager user=user2 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user2 component=silences msg="Maintenance done" duration=528.189µs size=0 --- PASS: TestMultitenantAlertmanager_zoneAwareSharding (1.07s) === RUN TestMultitenantAlertmanager_deleteUnusedRemoteUserState level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=instance-1 ring=alertmanager level=info component=MultiTenantAlertmanager msg="ring doesn't exist in KV store yet" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=instance-2 ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=2 level=debug component=MultiTenantAlertmanager msg="setting config" user=user1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user1 level=debug component=MultiTenantAlertmanager user=user1 component=nflog msg="notification log snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_deleteUnusedRemoteUserState966531061/001/user1/notifications: no such file or directory" level=debug component=MultiTenantAlertmanager user=user1 component=silences msg="silences snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_deleteUnusedRemoteUserState966531061/001/user1/silences: no such file or directory" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="no state for user in storage; proceeding" level=debug component=MultiTenantAlertmanager msg="setting config" user=user2 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user2 level=debug component=MultiTenantAlertmanager user=user2 component=nflog msg="notification log snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_deleteUnusedRemoteUserState966531061/001/user2/notifications: no such file or directory" level=debug component=MultiTenantAlertmanager user=user2 component=silences msg="silences snapshot file doesn't exist" err="open /tmp/TestMultitenantAlertmanager_deleteUnusedRemoteUserState966531061/001/user2/silences: no such file or directory" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=debug component=MultiTenantAlertmanager user=user1 msg="persisting state" user=user1 level=debug component=MultiTenantAlertmanager user=user2 msg="persisting state" user=user2 level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=2 level=debug component=MultiTenantAlertmanager msg="setting config" user=user1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user2 level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user2 level=info component=MultiTenantAlertmanager msg="deactivating per-tenant alertmanager" user=user1 level=debug component=MultiTenantAlertmanager user=user1 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user1 component=nflog msg="Maintenance done" duration=526.109µs size=0 level=debug component=MultiTenantAlertmanager user=user1 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user1 component=silences msg="Maintenance done" duration=321.458µs size=0 level=info component=MultiTenantAlertmanager msg="deactivated per-tenant alertmanager" user=user1 level=info component=MultiTenantAlertmanager msg="deleted local directory for user" dir=/tmp/TestMultitenantAlertmanager_deleteUnusedRemoteUserState966531061/001/user1 user=user1 level=info component=MultiTenantAlertmanager msg="deleted remote state for user" user=user1 level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="ring lifecycler is shutting down" ring=alertmanager level=info component=MultiTenantAlertmanager msg="unregistering instance from ring" ring=alertmanager level=info component=MultiTenantAlertmanager msg="instance removed from the ring" ring=alertmanager level=debug component=MultiTenantAlertmanager user=user2 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user2 component=nflog msg="Maintenance done" duration=489.946µs size=0 level=debug component=MultiTenantAlertmanager user=user2 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user2 component=silences msg="Maintenance done" duration=315.697µs size=0 level=info component=MultiTenantAlertmanager msg="ring lifecycler is shutting down" ring=alertmanager level=info component=MultiTenantAlertmanager msg="unregistering instance from ring" ring=alertmanager level=info component=MultiTenantAlertmanager msg="instance removed from the ring" ring=alertmanager --- PASS: TestMultitenantAlertmanager_deleteUnusedRemoteUserState (0.87s) === RUN TestMultitenantAlertmanager_ServeHTTP --- PASS: TestMultitenantAlertmanager_ServeHTTP (0.34s) === RUN TestMultitenantAlertmanager_ServeHTTPWithFallbackConfig --- PASS: TestMultitenantAlertmanager_ServeHTTPWithFallbackConfig (0.28s) === RUN TestMultitenantAlertmanager_ServeHTTPBeforeSyncFailsIfConfigExists --- PASS: TestMultitenantAlertmanager_ServeHTTPBeforeSyncFailsIfConfigExists (0.27s) === RUN TestMultitenantAlertmanager_InitialSync === RUN TestMultitenantAlertmanager_InitialSync/with_no_instance_in_the_ring === RUN TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_PENDING_state_and_no_tokens === RUN TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_JOINING_state_and_some_tokens === RUN TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_ACTIVE_state_and_all_tokens === RUN TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_LEAVING_state_and_all_tokens --- PASS: TestMultitenantAlertmanager_InitialSync (0.35s) --- PASS: TestMultitenantAlertmanager_InitialSync/with_no_instance_in_the_ring (0.01s) --- PASS: TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_PENDING_state_and_no_tokens (0.17s) --- PASS: TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_JOINING_state_and_some_tokens (0.13s) --- PASS: TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_ACTIVE_state_and_all_tokens (0.02s) --- PASS: TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_LEAVING_state_and_all_tokens (0.01s) === RUN TestMultitenantAlertmanager_PerTenantSharding === RUN TestMultitenantAlertmanager_PerTenantSharding/1_instance,_RF_=_1 === RUN TestMultitenantAlertmanager_PerTenantSharding/2_instances,_RF_=_1 === RUN TestMultitenantAlertmanager_PerTenantSharding/3_instances,_RF_=_2 === RUN TestMultitenantAlertmanager_PerTenantSharding/5_instances,_RF_=_3 --- PASS: TestMultitenantAlertmanager_PerTenantSharding (10.31s) --- PASS: TestMultitenantAlertmanager_PerTenantSharding/1_instance,_RF_=_1 (1.17s) --- PASS: TestMultitenantAlertmanager_PerTenantSharding/2_instances,_RF_=_1 (1.66s) --- PASS: TestMultitenantAlertmanager_PerTenantSharding/3_instances,_RF_=_2 (3.07s) --- PASS: TestMultitenantAlertmanager_PerTenantSharding/5_instances,_RF_=_3 (4.40s) === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/when_an_instance_is_added_to_the_ring === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/when_an_instance_is_removed_from_the_ring === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_sync_when_an_instance_changes_state === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_sync_when_an_healthy_instance_becomes_unhealthy === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_sync_when_an_unhealthy_instance_becomes_healthy === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_NOT_sync_when_an_instance_updates_the_heartbeat === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_NOT_sync_when_an_instance_is_auto-forgotten_in_the_ring_but_was_already_unhealthy_in_the_previous_state --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges (2.16s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/when_an_instance_is_added_to_the_ring (0.27s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/when_an_instance_is_removed_from_the_ring (0.48s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_sync_when_an_instance_changes_state (0.51s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_sync_when_an_healthy_instance_becomes_unhealthy (0.32s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_sync_when_an_unhealthy_instance_becomes_healthy (0.36s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_NOT_sync_when_an_instance_updates_the_heartbeat (0.14s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_NOT_sync_when_an_instance_is_auto-forgotten_in_the_ring_but_was_already_unhealthy_in_the_previous_state (0.05s) === RUN TestMultitenantAlertmanager_RingLifecyclerShouldAutoForgetUnhealthyInstances --- PASS: TestMultitenantAlertmanager_RingLifecyclerShouldAutoForgetUnhealthyInstances (0.22s) === RUN TestMultitenantAlertmanager_InitialSyncFailure --- PASS: TestMultitenantAlertmanager_InitialSyncFailure (0.16s) === RUN TestAlertmanager_ReplicasPosition --- PASS: TestAlertmanager_ReplicasPosition (0.73s) === RUN TestAlertmanager_StateReplication === RUN TestAlertmanager_StateReplication/RF_=_1,_1_instance === RUN TestAlertmanager_StateReplication/RF_=_2,_2_instances === RUN TestAlertmanager_StateReplication/RF_=_3,_10_instance --- PASS: TestAlertmanager_StateReplication (13.99s) --- PASS: TestAlertmanager_StateReplication/RF_=_1,_1_instance (1.36s) --- PASS: TestAlertmanager_StateReplication/RF_=_2,_2_instances (3.05s) --- PASS: TestAlertmanager_StateReplication/RF_=_3,_10_instance (9.56s) === RUN TestAlertmanager_StateReplication_InitialSyncFromPeers === RUN TestAlertmanager_StateReplication_InitialSyncFromPeers/RF_=_2 === RUN TestAlertmanager_StateReplication_InitialSyncFromPeers/RF_=_3 --- PASS: TestAlertmanager_StateReplication_InitialSyncFromPeers (1.54s) --- PASS: TestAlertmanager_StateReplication_InitialSyncFromPeers/RF_=_2 (0.38s) --- PASS: TestAlertmanager_StateReplication_InitialSyncFromPeers/RF_=_3 (1.15s) === RUN TestSafeTemplateFilepath === RUN TestSafeTemplateFilepath/should_fail_if_the_provided_template_is_escaping_the_dir === RUN TestSafeTemplateFilepath/escaping_template_name_that_has_prefix_of_dir_(tmp_is_prefix_of_tmpfile) === RUN TestSafeTemplateFilepath/empty_template_name === RUN TestSafeTemplateFilepath/dot_template_name === RUN TestSafeTemplateFilepath/should_succeed_if_the_provided_template_is_a_filename === RUN TestSafeTemplateFilepath/template_name_starting_with_/ === RUN TestSafeTemplateFilepath/root_dir === RUN TestSafeTemplateFilepath/root_dir_2 --- PASS: TestSafeTemplateFilepath (0.00s) --- PASS: TestSafeTemplateFilepath/should_fail_if_the_provided_template_is_escaping_the_dir (0.00s) --- PASS: TestSafeTemplateFilepath/escaping_template_name_that_has_prefix_of_dir_(tmp_is_prefix_of_tmpfile) (0.00s) --- PASS: TestSafeTemplateFilepath/empty_template_name (0.00s) --- PASS: TestSafeTemplateFilepath/dot_template_name (0.00s) --- PASS: TestSafeTemplateFilepath/should_succeed_if_the_provided_template_is_a_filename (0.00s) --- PASS: TestSafeTemplateFilepath/template_name_starting_with_/ (0.00s) --- PASS: TestSafeTemplateFilepath/root_dir (0.00s) --- PASS: TestSafeTemplateFilepath/root_dir_2 (0.00s) === RUN TestStoreTemplateFile --- PASS: TestStoreTemplateFile (0.00s) === RUN TestMultitenantAlertmanager_verifyRateLimitedEmailConfig --- PASS: TestMultitenantAlertmanager_verifyRateLimitedEmailConfig (0.19s) === RUN TestMultitenantAlertmanager_computeFallbackConfig --- PASS: TestMultitenantAlertmanager_computeFallbackConfig (0.00s) === RUN TestRateLimitedNotifier --- PASS: TestRateLimitedNotifier (1.00s) === RUN TestStatePersister_Position0ShouldWrite --- PASS: TestStatePersister_Position0ShouldWrite (6.10s) === RUN TestStatePersister_Position1ShouldNotWrite --- PASS: TestStatePersister_Position1ShouldNotWrite (5.00s) === RUN TestStateReplication === RUN TestStateReplication/with_a_replication_factor_of_<=_1,_state_is_not_replicated_but_loaded_from_storage. === RUN TestStateReplication/with_a_replication_factor_of_>_1,_state_is_broadcasted_for_replication. --- PASS: TestStateReplication (0.21s) --- PASS: TestStateReplication/with_a_replication_factor_of_<=_1,_state_is_not_replicated_but_loaded_from_storage. (0.11s) --- PASS: TestStateReplication/with_a_replication_factor_of_>_1,_state_is_broadcasted_for_replication. (0.11s) === RUN TestStateReplication_Settle === RUN TestStateReplication_Settle/with_a_replication_factor_of_<=_1,_no_state_can_be_read_from_peers. === RUN TestStateReplication_Settle/with_a_replication_factor_of_>_1,_state_is_read_from_all_peers. === RUN TestStateReplication_Settle/with_full_state_having_no_parts,_nothing_is_merged. === RUN TestStateReplication_Settle/with_an_unknown_key,_parts_in_the_same_state_are_merged. === RUN TestStateReplication_Settle/with_an_unknown_key,_parts_in_other_states_are_merged. === RUN TestStateReplication_Settle/when_reading_from_replicas_fails,_state_is_read_from_storage. === RUN TestStateReplication_Settle/when_reading_from_replicas_and_from_storage_fails,_still_become_ready. === RUN TestStateReplication_Settle/when_user_not_found_in_all_replicas_and_storage,_read_not_counted_as_failure_and_still_become_ready. === RUN TestStateReplication_Settle/when_reading_the_full_state_takes_too_long,_hit_timeout_but_become_ready. --- PASS: TestStateReplication_Settle (1.03s) --- PASS: TestStateReplication_Settle/with_a_replication_factor_of_<=_1,_no_state_can_be_read_from_peers. (0.00s) --- PASS: TestStateReplication_Settle/with_a_replication_factor_of_>_1,_state_is_read_from_all_peers. (0.00s) --- PASS: TestStateReplication_Settle/with_full_state_having_no_parts,_nothing_is_merged. (0.00s) --- PASS: TestStateReplication_Settle/with_an_unknown_key,_parts_in_the_same_state_are_merged. (0.00s) --- PASS: TestStateReplication_Settle/with_an_unknown_key,_parts_in_other_states_are_merged. (0.00s) --- PASS: TestStateReplication_Settle/when_reading_from_replicas_fails,_state_is_read_from_storage. (0.00s) --- PASS: TestStateReplication_Settle/when_reading_from_replicas_and_from_storage_fails,_still_become_ready. (0.00s) --- PASS: TestStateReplication_Settle/when_user_not_found_in_all_replicas_and_storage,_read_not_counted_as_failure_and_still_become_ready. (0.00s) --- PASS: TestStateReplication_Settle/when_reading_the_full_state_takes_too_long,_hit_timeout_but_become_ready. (1.00s) === RUN TestStateReplication_GetFullState === RUN TestStateReplication_GetFullState/no_keys === RUN TestStateReplication_GetFullState/zero_length_data === RUN TestStateReplication_GetFullState/keys_with_data --- PASS: TestStateReplication_GetFullState (0.00s) --- PASS: TestStateReplication_GetFullState/no_keys (0.00s) --- PASS: TestStateReplication_GetFullState/zero_length_data (0.00s) --- PASS: TestStateReplication_GetFullState/keys_with_data (0.00s) === CONT TestIsHealthyForAlertmanagerOperations === RUN TestIsHealthyForAlertmanagerOperations/ACTIVE_instance_with_last_keepalive_newer_than_timeout === RUN TestIsHealthyForAlertmanagerOperations/ACTIVE_instance_with_last_keepalive_older_than_timeout === RUN TestIsHealthyForAlertmanagerOperations/JOINING_instance_with_last_keepalive_newer_than_timeout === RUN TestIsHealthyForAlertmanagerOperations/LEAVING_instance_with_last_keepalive_newer_than_timeout === RUN TestIsHealthyForAlertmanagerOperations/PENDING_instance_with_last_keepalive_newer_than_timeout --- PASS: TestIsHealthyForAlertmanagerOperations (0.00s) --- PASS: TestIsHealthyForAlertmanagerOperations/ACTIVE_instance_with_last_keepalive_newer_than_timeout (0.00s) --- PASS: TestIsHealthyForAlertmanagerOperations/ACTIVE_instance_with_last_keepalive_older_than_timeout (0.00s) --- PASS: TestIsHealthyForAlertmanagerOperations/JOINING_instance_with_last_keepalive_newer_than_timeout (0.00s) --- PASS: TestIsHealthyForAlertmanagerOperations/LEAVING_instance_with_last_keepalive_newer_than_timeout (0.00s) --- PASS: TestIsHealthyForAlertmanagerOperations/PENDING_instance_with_last_keepalive_newer_than_timeout (0.00s) PASS ok github.com/grafana/mimir/pkg/alertmanager 69.063s === RUN TestAlertStore_ListAllUsers --- PASS: TestAlertStore_ListAllUsers (0.00s) === RUN TestAlertStore_SetAndGetAlertConfig --- PASS: TestAlertStore_SetAndGetAlertConfig (0.00s) === RUN TestStore_GetAlertConfigs --- PASS: TestStore_GetAlertConfigs (0.00s) === RUN TestAlertStore_DeleteAlertConfig --- PASS: TestAlertStore_DeleteAlertConfig (0.00s) === RUN TestBucketAlertStore_GetSetDeleteFullState --- PASS: TestBucketAlertStore_GetSetDeleteFullState (0.00s) PASS ok github.com/grafana/mimir/pkg/alertmanager/alertstore 0.337s testing: warning: no tests to run PASS ok github.com/grafana/mimir/pkg/alertmanager/alertstore/bucketclient 0.442s [no tests to run] === RUN TestStore_ListAllUsers --- PASS: TestStore_ListAllUsers (0.01s) === RUN TestStore_GetAlertConfig --- PASS: TestStore_GetAlertConfig (0.02s) === RUN TestStore_GetAlertConfigs --- PASS: TestStore_GetAlertConfigs (0.03s) === RUN TestStore_FullState --- PASS: TestStore_FullState (0.00s) PASS ok github.com/grafana/mimir/pkg/alertmanager/alertstore/local 0.189s === RUN TestV1Alerts --- PASS: TestV1Alerts (0.00s) === RUN TestMergeV1Alerts === RUN TestMergeV1Alerts/no_alerts,_should_return_an_empty_list === RUN TestMergeV1Alerts/one_alert,_should_return_the_alert === RUN TestMergeV1Alerts/two_alerts,_should_return_two_alerts === RUN TestMergeV1Alerts/three_alerts,_should_return_three_alerts === RUN TestMergeV1Alerts/three_alerts_out_of_order,_should_return_three_alerts_in_fingerprint_order === RUN TestMergeV1Alerts/two_identical_alerts,_should_return_one_alert === RUN TestMergeV1Alerts/two_identical_alerts_plus_another,_should_return_two_alerts === RUN TestMergeV1Alerts/two_duplicates_out_of_sync_alerts,_should_return_first_seen_alert === RUN TestMergeV1Alerts/two_duplicates_plus_others,_should_return_first_seen_alert_and_others --- PASS: TestMergeV1Alerts (0.00s) --- PASS: TestMergeV1Alerts/no_alerts,_should_return_an_empty_list (0.00s) --- PASS: TestMergeV1Alerts/one_alert,_should_return_the_alert (0.00s) --- PASS: TestMergeV1Alerts/two_alerts,_should_return_two_alerts (0.00s) --- PASS: TestMergeV1Alerts/three_alerts,_should_return_three_alerts (0.00s) --- PASS: TestMergeV1Alerts/three_alerts_out_of_order,_should_return_three_alerts_in_fingerprint_order (0.00s) --- PASS: TestMergeV1Alerts/two_identical_alerts,_should_return_one_alert (0.00s) --- PASS: TestMergeV1Alerts/two_identical_alerts_plus_another,_should_return_two_alerts (0.00s) --- PASS: TestMergeV1Alerts/two_duplicates_out_of_sync_alerts,_should_return_first_seen_alert (0.00s) --- PASS: TestMergeV1Alerts/two_duplicates_plus_others,_should_return_first_seen_alert_and_others (0.00s) === RUN TestV1SilenceID_ReturnsNewestSilence --- PASS: TestV1SilenceID_ReturnsNewestSilence (0.00s) === RUN TestV1SilenceID_InvalidDifferentIDs --- PASS: TestV1SilenceID_InvalidDifferentIDs (0.00s) === RUN TestV1Silences --- PASS: TestV1Silences (0.00s) === RUN TestV2AlertGroups --- PASS: TestV2AlertGroups (0.00s) === RUN TestMergeV2AlertGroups === RUN TestMergeV2AlertGroups/no_groups,_should_return_no_groups === RUN TestMergeV2AlertGroups/one_group_with_one_alert,_should_return_one_group === RUN TestMergeV2AlertGroups/two_groups_with_different_labels,_should_return_two_groups === RUN TestMergeV2AlertGroups/two_groups_with_different_receiver,_should_return_two_groups === RUN TestMergeV2AlertGroups/two_identical_groups_with_different_alerts,_should_return_one_group_with_two_alerts === RUN TestMergeV2AlertGroups/two_identical_groups_with_identical_alerts,_should_return_one_group_with_one_alert === RUN TestMergeV2AlertGroups/two_identical_groups_with_diverged_alerts,_should_return_one_group_with_the_newer_alert === RUN TestMergeV2AlertGroups/two_sets_of_identical_groups_with_single_alerts,_should_merge_all_into_two_groups_ === RUN TestMergeV2AlertGroups/many_unordered_groups,_should_return_groups_ordered_by_labels_then_receiver --- PASS: TestMergeV2AlertGroups (0.01s) --- PASS: TestMergeV2AlertGroups/no_groups,_should_return_no_groups (0.00s) --- PASS: TestMergeV2AlertGroups/one_group_with_one_alert,_should_return_one_group (0.00s) --- PASS: TestMergeV2AlertGroups/two_groups_with_different_labels,_should_return_two_groups (0.00s) --- PASS: TestMergeV2AlertGroups/two_groups_with_different_receiver,_should_return_two_groups (0.00s) --- PASS: TestMergeV2AlertGroups/two_identical_groups_with_different_alerts,_should_return_one_group_with_two_alerts (0.00s) --- PASS: TestMergeV2AlertGroups/two_identical_groups_with_identical_alerts,_should_return_one_group_with_one_alert (0.00s) --- PASS: TestMergeV2AlertGroups/two_identical_groups_with_diverged_alerts,_should_return_one_group_with_the_newer_alert (0.00s) --- PASS: TestMergeV2AlertGroups/two_sets_of_identical_groups_with_single_alerts,_should_merge_all_into_two_groups_ (0.00s) --- PASS: TestMergeV2AlertGroups/many_unordered_groups,_should_return_groups_ordered_by_labels_then_receiver (0.00s) === RUN TestV2Alerts --- PASS: TestV2Alerts (0.00s) === RUN TestMergeV2Alerts === RUN TestMergeV2Alerts/no_alerts,_should_return_an_empty_list === RUN TestMergeV2Alerts/one_alert,_should_return_the_alert === RUN TestMergeV2Alerts/two_alerts,_should_return_two_alerts === RUN TestMergeV2Alerts/three_alerts,_should_return_three_alerts === RUN TestMergeV2Alerts/three_alerts_out_of_order,_should_return_three_alerts_in_fingerprint_order === RUN TestMergeV2Alerts/two_identical_alerts,_should_return_one_alert === RUN TestMergeV2Alerts/two_identical_alerts_plus_another,_should_return_two_alerts === RUN TestMergeV2Alerts/two_duplicates_out_of_sync_alerts,_should_return_newer_alert === RUN TestMergeV2Alerts/two_duplicates_out_of_sync_alerts_(newer_first),_should_return_newer_alert === RUN TestMergeV2Alerts/two_duplicates_plus_others,_should_return_newer_alert_and_others --- PASS: TestMergeV2Alerts (0.01s) --- PASS: TestMergeV2Alerts/no_alerts,_should_return_an_empty_list (0.00s) --- PASS: TestMergeV2Alerts/one_alert,_should_return_the_alert (0.00s) --- PASS: TestMergeV2Alerts/two_alerts,_should_return_two_alerts (0.00s) --- PASS: TestMergeV2Alerts/three_alerts,_should_return_three_alerts (0.00s) --- PASS: TestMergeV2Alerts/three_alerts_out_of_order,_should_return_three_alerts_in_fingerprint_order (0.00s) --- PASS: TestMergeV2Alerts/two_identical_alerts,_should_return_one_alert (0.00s) --- PASS: TestMergeV2Alerts/two_identical_alerts_plus_another,_should_return_two_alerts (0.00s) --- PASS: TestMergeV2Alerts/two_duplicates_out_of_sync_alerts,_should_return_newer_alert (0.00s) --- PASS: TestMergeV2Alerts/two_duplicates_out_of_sync_alerts_(newer_first),_should_return_newer_alert (0.00s) --- PASS: TestMergeV2Alerts/two_duplicates_plus_others,_should_return_newer_alert_and_others (0.00s) === RUN TestV2SilenceId_ReturnsNewestSilence --- PASS: TestV2SilenceId_ReturnsNewestSilence (0.00s) === RUN TestV2SilenceID_InvalidDifferentIDs --- PASS: TestV2SilenceID_InvalidDifferentIDs (0.00s) === RUN TestV2Silences --- PASS: TestV2Silences (0.00s) === RUN TestMergeV2Silences === RUN TestMergeV2Silences/no_silences,_should_return_an_empty_list === RUN TestMergeV2Silences/one_silence,_should_return_the_silence === RUN TestMergeV2Silences/two_silences,_should_return_two_silences === RUN TestMergeV2Silences/three_silences,_should_return_three_silences === RUN TestMergeV2Silences/three_active_silences_out_of_order,_should_return_three_silences_in_expiry_order === RUN TestMergeV2Silences/two_identical_silences,_should_return_one_silence === RUN TestMergeV2Silences/two_identical_silences_plus_another,_should_return_two_silences === RUN TestMergeV2Silences/two_duplicates_out_of_sync_silences,_should_return_newer_silence === RUN TestMergeV2Silences/two_duplicates_out_of_sync_silences_(newer_first),_should_return_newer_silence === RUN TestMergeV2Silences/two_duplicates_plus_others,_should_return_newer_silence_and_others --- PASS: TestMergeV2Silences (0.01s) --- PASS: TestMergeV2Silences/no_silences,_should_return_an_empty_list (0.00s) --- PASS: TestMergeV2Silences/one_silence,_should_return_the_silence (0.00s) --- PASS: TestMergeV2Silences/two_silences,_should_return_two_silences (0.00s) --- PASS: TestMergeV2Silences/three_silences,_should_return_three_silences (0.00s) --- PASS: TestMergeV2Silences/three_active_silences_out_of_order,_should_return_three_silences_in_expiry_order (0.00s) --- PASS: TestMergeV2Silences/two_identical_silences,_should_return_one_silence (0.00s) --- PASS: TestMergeV2Silences/two_identical_silences_plus_another,_should_return_two_silences (0.00s) --- PASS: TestMergeV2Silences/two_duplicates_out_of_sync_silences,_should_return_newer_silence (0.00s) --- PASS: TestMergeV2Silences/two_duplicates_out_of_sync_silences_(newer_first),_should_return_newer_silence (0.00s) --- PASS: TestMergeV2Silences/two_duplicates_plus_others,_should_return_newer_silence_and_others (0.00s) PASS ok github.com/grafana/mimir/pkg/alertmanager/merger 0.249s === RUN TestNewApiWithoutSourceIPExtractor --- PASS: TestNewApiWithoutSourceIPExtractor (0.01s) === RUN TestNewApiWithSourceIPExtractor --- PASS: TestNewApiWithSourceIPExtractor (0.00s) === RUN TestNewApiWithInvalidSourceIPExtractor --- PASS: TestNewApiWithInvalidSourceIPExtractor (0.00s) === RUN TestApiGzip === RUN TestApiGzip/happy_case_gzip === RUN TestApiGzip/gzip_with_priority_header === RUN TestApiGzip/gzip_because_any_is_accepted === RUN TestApiGzip/no_gzip_because_no_header === RUN TestApiGzip/no_gzip_because_not_accepted === RUN TestApiGzip/no_gzip_because_small_payload === RUN TestApiGzip/forced_gzip_with_small_payload === RUN TestApiGzip/gzip_disabled_endpoint === RUN TestApiGzip/compressed_with_gzip --- PASS: TestApiGzip (0.06s) --- PASS: TestApiGzip/happy_case_gzip (0.01s) --- PASS: TestApiGzip/gzip_with_priority_header (0.01s) --- PASS: TestApiGzip/gzip_because_any_is_accepted (0.01s) --- PASS: TestApiGzip/no_gzip_because_no_header (0.01s) --- PASS: TestApiGzip/no_gzip_because_not_accepted (0.00s) --- PASS: TestApiGzip/no_gzip_because_small_payload (0.00s) --- PASS: TestApiGzip/forced_gzip_with_small_payload (0.01s) --- PASS: TestApiGzip/gzip_disabled_endpoint (0.00s) --- PASS: TestApiGzip/compressed_with_gzip (0.00s) === RUN TestIndexHandlerPrefix --- PASS: TestIndexHandlerPrefix (0.01s) === RUN TestIndexPageContent --- PASS: TestIndexPageContent (0.00s) === RUN TestConfigDiffHandler === RUN TestConfigDiffHandler/no_config_parameters_overridden === RUN TestConfigDiffHandler/slice_changed === RUN TestConfigDiffHandler/string_in_nested_struct_changed === RUN TestConfigDiffHandler/bool_in_nested_struct_changed === RUN TestConfigDiffHandler/test_invalid_input --- PASS: TestConfigDiffHandler (0.02s) --- PASS: TestConfigDiffHandler/no_config_parameters_overridden (0.00s) --- PASS: TestConfigDiffHandler/slice_changed (0.00s) --- PASS: TestConfigDiffHandler/string_in_nested_struct_changed (0.00s) --- PASS: TestConfigDiffHandler/bool_in_nested_struct_changed (0.00s) --- PASS: TestConfigDiffHandler/test_invalid_input (0.01s) === RUN TestConfigOverrideHandler --- PASS: TestConfigOverrideHandler (0.00s) === RUN TestProtobufCodec_CanEncode === RUN TestProtobufCodec_CanEncode/another_type_of_result === RUN TestProtobufCodec_CanEncode/query_results === RUN TestProtobufCodec_CanEncode/error --- PASS: TestProtobufCodec_CanEncode (0.00s) --- PASS: TestProtobufCodec_CanEncode/another_type_of_result (0.00s) --- PASS: TestProtobufCodec_CanEncode/query_results (0.00s) --- PASS: TestProtobufCodec_CanEncode/error (0.00s) === RUN TestProtobufCodec_Encode === RUN TestProtobufCodec_Encode/vector_with_single_series_with_many_labels === RUN TestProtobufCodec_Encode/vector_with_multiple_series === RUN TestProtobufCodec_Encode/vector_with_float_and_histogram_values === RUN TestProtobufCodec_Encode/error === RUN TestProtobufCodec_Encode/scalar === RUN TestProtobufCodec_Encode/matrix_with_no_series === RUN TestProtobufCodec_Encode/matrix_with_single_series_with_one_point === RUN TestProtobufCodec_Encode/matrix_with_many_series === RUN TestProtobufCodec_Encode/matrix_with_series_with_histogram_value === RUN TestProtobufCodec_Encode/vector_with_single_series_with_one_label === RUN TestProtobufCodec_Encode/vector_with_series_with_histogram_value === RUN TestProtobufCodec_Encode/matrix_with_single_series_with_many_points === RUN TestProtobufCodec_Encode/vector_with_single_series_with_no_labels === RUN TestProtobufCodec_Encode/matrix_with_single_series_with_no_points_and_many_labels === RUN TestProtobufCodec_Encode/matrix_with_single_series_with_no_points_and_no_labels === RUN TestProtobufCodec_Encode/matrix_with_single_series_with_no_points_and_one_label === RUN TestProtobufCodec_Encode/matrix_with_series_with_both_float_and_histogram_values === RUN TestProtobufCodec_Encode/string === RUN TestProtobufCodec_Encode/vector_with_no_series --- PASS: TestProtobufCodec_Encode (0.04s) --- PASS: TestProtobufCodec_Encode/vector_with_single_series_with_many_labels (0.00s) --- PASS: TestProtobufCodec_Encode/vector_with_multiple_series (0.00s) --- PASS: TestProtobufCodec_Encode/vector_with_float_and_histogram_values (0.00s) --- PASS: TestProtobufCodec_Encode/error (0.00s) --- PASS: TestProtobufCodec_Encode/scalar (0.01s) --- PASS: TestProtobufCodec_Encode/matrix_with_no_series (0.00s) --- PASS: TestProtobufCodec_Encode/matrix_with_single_series_with_one_point (0.00s) --- PASS: TestProtobufCodec_Encode/matrix_with_many_series (0.00s) --- PASS: TestProtobufCodec_Encode/matrix_with_series_with_histogram_value (0.00s) --- PASS: TestProtobufCodec_Encode/vector_with_single_series_with_one_label (0.00s) --- PASS: TestProtobufCodec_Encode/vector_with_series_with_histogram_value (0.00s) --- PASS: TestProtobufCodec_Encode/matrix_with_single_series_with_many_points (0.00s) --- PASS: TestProtobufCodec_Encode/vector_with_single_series_with_no_labels (0.00s) --- PASS: TestProtobufCodec_Encode/matrix_with_single_series_with_no_points_and_many_labels (0.00s) --- PASS: TestProtobufCodec_Encode/matrix_with_single_series_with_no_points_and_no_labels (0.01s) --- PASS: TestProtobufCodec_Encode/matrix_with_single_series_with_no_points_and_one_label (0.00s) --- PASS: TestProtobufCodec_Encode/matrix_with_series_with_both_float_and_histogram_values (0.00s) --- PASS: TestProtobufCodec_Encode/string (0.00s) --- PASS: TestProtobufCodec_Encode/vector_with_no_series (0.00s) PASS ok github.com/grafana/mimir/pkg/api 0.710s === RUN TestAllPrometheusErrorTypeValues --- PASS: TestAllPrometheusErrorTypeValues (0.04s) PASS ok github.com/grafana/mimir/pkg/api/error 0.128s ? github.com/grafana/mimir/pkg/distributor/distributorpb [no test files] === RUN TestMultitenantCompactor_StartBlockUpload === RUN TestMultitenantCompactor_StartBlockUpload/missing_tenant_ID === RUN TestMultitenantCompactor_StartBlockUpload/missing_block_ID === RUN TestMultitenantCompactor_StartBlockUpload/invalid_block_ID === RUN TestMultitenantCompactor_StartBlockUpload/missing_body block_upload_test.go:600: PASS: Exists(string,string) block_upload_test.go:600: PASS: Get(string,string) === RUN TestMultitenantCompactor_StartBlockUpload/malformed_body block_upload_test.go:600: PASS: Exists(string,string) block_upload_test.go:600: PASS: Get(string,string) === RUN TestMultitenantCompactor_StartBlockUpload/invalid_file_path block_upload_test.go:600: PASS: Exists(string,string) block_upload_test.go:600: PASS: Get(string,string) === RUN TestMultitenantCompactor_StartBlockUpload/contains_downsampled_data block_upload_test.go:600: PASS: Exists(string,string) block_upload_test.go:600: PASS: Get(string,string) === RUN TestMultitenantCompactor_StartBlockUpload/missing_file_size block_upload_test.go:600: PASS: Exists(string,string) block_upload_test.go:600: PASS: Get(string,string) === RUN TestMultitenantCompactor_StartBlockUpload/invalid_minTime block_upload_test.go:600: PASS: Exists(string,string) block_upload_test.go:600: PASS: Get(string,string) === RUN TestMultitenantCompactor_StartBlockUpload/invalid_maxTime block_upload_test.go:600: PASS: Exists(string,string) block_upload_test.go:600: PASS: Get(string,string) === RUN TestMultitenantCompactor_StartBlockUpload/maxTime_before_minTime block_upload_test.go:600: PASS: Exists(string,string) block_upload_test.go:600: PASS: Get(string,string) === RUN TestMultitenantCompactor_StartBlockUpload/block_before_retention_period block_upload_test.go:600: PASS: Exists(string,string) block_upload_test.go:600: PASS: Get(string,string) === RUN TestMultitenantCompactor_StartBlockUpload/invalid_version block_upload_test.go:600: PASS: Exists(string,string) block_upload_test.go:600: PASS: Get(string,string) === RUN TestMultitenantCompactor_StartBlockUpload/ignore_retention_period_if_==_0 block_upload_test.go:600: PASS: Exists(string,string) block_upload_test.go:600: PASS: Get(string,string) block_upload_test.go:600: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_StartBlockUpload/ignore_retention_period_if_<_0 block_upload_test.go:600: PASS: Exists(string,string) block_upload_test.go:600: PASS: Get(string,string) block_upload_test.go:600: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_StartBlockUpload/invalid_compactor_shard_ID_label block_upload_test.go:600: PASS: Exists(string,string) block_upload_test.go:600: PASS: Get(string,string) === RUN TestMultitenantCompactor_StartBlockUpload/failure_checking_for_complete_block block_upload_test.go:600: PASS: Exists(string,string) === RUN TestMultitenantCompactor_StartBlockUpload/complete_block_already_exists block_upload_test.go:600: PASS: Exists(string,string) === RUN TestMultitenantCompactor_StartBlockUpload/failure_uploading_meta_file block_upload_test.go:600: PASS: Exists(string,string) block_upload_test.go:600: PASS: Get(string,string) block_upload_test.go:600: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_StartBlockUpload/too_large_of_a_request_body block_upload_test.go:600: PASS: Exists(string,string) block_upload_test.go:600: PASS: Get(string,string) === RUN TestMultitenantCompactor_StartBlockUpload/block_upload_disabled === RUN TestMultitenantCompactor_StartBlockUpload/valid_request block_upload_test.go:600: PASS: Exists(string,string) block_upload_test.go:600: PASS: Get(string,string) block_upload_test.go:600: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_StartBlockUpload/valid_request_with_empty_compactor_shard_ID_label block_upload_test.go:600: PASS: Exists(string,string) block_upload_test.go:600: PASS: Get(string,string) block_upload_test.go:600: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_StartBlockUpload/valid_request_without_compactor_shard_ID_label block_upload_test.go:600: PASS: Exists(string,string) block_upload_test.go:600: PASS: Get(string,string) block_upload_test.go:600: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_StartBlockUpload/valid_request_with_different_block_ID_in_meta_file block_upload_test.go:600: PASS: Exists(string,string) block_upload_test.go:600: PASS: Get(string,string) block_upload_test.go:600: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_StartBlockUpload/valid_request_when_both_in-flight_meta_file_and_complete_meta_file_exist_in_object_storage === RUN TestMultitenantCompactor_StartBlockUpload/invalid_request_when_in-flight_meta_file_exists_in_object_storage === RUN TestMultitenantCompactor_StartBlockUpload/valid_request_when_same_in-flight_meta_file_exists_in_object_storage === RUN TestMultitenantCompactor_StartBlockUpload/valid_request_when_different_in-flight_meta_file_exists_in_object_storage --- PASS: TestMultitenantCompactor_StartBlockUpload (0.35s) --- PASS: TestMultitenantCompactor_StartBlockUpload/missing_tenant_ID (0.00s) --- PASS: TestMultitenantCompactor_StartBlockUpload/missing_block_ID (0.01s) --- PASS: TestMultitenantCompactor_StartBlockUpload/invalid_block_ID (0.00s) --- PASS: TestMultitenantCompactor_StartBlockUpload/missing_body (0.02s) --- PASS: TestMultitenantCompactor_StartBlockUpload/malformed_body (0.00s) --- PASS: TestMultitenantCompactor_StartBlockUpload/invalid_file_path (0.01s) --- PASS: TestMultitenantCompactor_StartBlockUpload/contains_downsampled_data (0.00s) --- PASS: TestMultitenantCompactor_StartBlockUpload/missing_file_size (0.00s) --- PASS: TestMultitenantCompactor_StartBlockUpload/invalid_minTime (0.00s) --- PASS: TestMultitenantCompactor_StartBlockUpload/invalid_maxTime (0.00s) --- PASS: TestMultitenantCompactor_StartBlockUpload/maxTime_before_minTime (0.00s) --- PASS: TestMultitenantCompactor_StartBlockUpload/block_before_retention_period (0.00s) --- PASS: TestMultitenantCompactor_StartBlockUpload/invalid_version (0.00s) --- PASS: TestMultitenantCompactor_StartBlockUpload/ignore_retention_period_if_==_0 (0.00s) --- PASS: TestMultitenantCompactor_StartBlockUpload/ignore_retention_period_if_<_0 (0.00s) --- PASS: TestMultitenantCompactor_StartBlockUpload/invalid_compactor_shard_ID_label (0.00s) --- PASS: TestMultitenantCompactor_StartBlockUpload/failure_checking_for_complete_block (0.00s) --- PASS: TestMultitenantCompactor_StartBlockUpload/complete_block_already_exists (0.00s) --- PASS: TestMultitenantCompactor_StartBlockUpload/failure_uploading_meta_file (0.00s) --- PASS: TestMultitenantCompactor_StartBlockUpload/too_large_of_a_request_body (0.23s) --- PASS: TestMultitenantCompactor_StartBlockUpload/block_upload_disabled (0.00s) --- PASS: TestMultitenantCompactor_StartBlockUpload/valid_request (0.01s) --- PASS: TestMultitenantCompactor_StartBlockUpload/valid_request_with_empty_compactor_shard_ID_label (0.01s) --- PASS: TestMultitenantCompactor_StartBlockUpload/valid_request_without_compactor_shard_ID_label (0.00s) --- PASS: TestMultitenantCompactor_StartBlockUpload/valid_request_with_different_block_ID_in_meta_file (0.00s) --- PASS: TestMultitenantCompactor_StartBlockUpload/valid_request_when_both_in-flight_meta_file_and_complete_meta_file_exist_in_object_storage (0.00s) --- PASS: TestMultitenantCompactor_StartBlockUpload/invalid_request_when_in-flight_meta_file_exists_in_object_storage (0.00s) --- PASS: TestMultitenantCompactor_StartBlockUpload/valid_request_when_same_in-flight_meta_file_exists_in_object_storage (0.00s) --- PASS: TestMultitenantCompactor_StartBlockUpload/valid_request_when_different_in-flight_meta_file_exists_in_object_storage (0.00s) === RUN TestMultitenantCompactor_UploadBlockFile === RUN TestMultitenantCompactor_UploadBlockFile/without_tenant_ID === RUN TestMultitenantCompactor_UploadBlockFile/without_block_ID === RUN TestMultitenantCompactor_UploadBlockFile/invalid_block_ID === RUN TestMultitenantCompactor_UploadBlockFile/without_path === RUN TestMultitenantCompactor_UploadBlockFile/invalid_path === RUN TestMultitenantCompactor_UploadBlockFile/empty_file === RUN TestMultitenantCompactor_UploadBlockFile/attempt_block_metadata_file === RUN TestMultitenantCompactor_UploadBlockFile/attempt_in-flight_block_metadata_file === RUN TestMultitenantCompactor_UploadBlockFile/block_upload_disabled === RUN TestMultitenantCompactor_UploadBlockFile/complete_block_already_exists block_upload_test.go:1012: PASS: Exists(string,string) === RUN TestMultitenantCompactor_UploadBlockFile/failure_checking_for_complete_block block_upload_test.go:1012: PASS: Exists(string,string) === RUN TestMultitenantCompactor_UploadBlockFile/failure_checking_for_in-flight_meta_file block_upload_test.go:1012: PASS: Exists(string,string) block_upload_test.go:1012: PASS: Get(string,string) === RUN TestMultitenantCompactor_UploadBlockFile/missing_in-flight_meta_file block_upload_test.go:1012: PASS: Exists(string,string) block_upload_test.go:1012: PASS: Get(string,string) === RUN TestMultitenantCompactor_UploadBlockFile/file_upload_fails block_upload_test.go:1012: PASS: Exists(string,string) block_upload_test.go:1012: PASS: Get(string,string) block_upload_test.go:1012: PASS: Get(string,string) block_upload_test.go:1012: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_UploadBlockFile/invalid_file_size block_upload_test.go:1012: PASS: Exists(string,string) block_upload_test.go:1012: PASS: Get(string,string) block_upload_test.go:1012: PASS: Get(string,string) === RUN TestMultitenantCompactor_UploadBlockFile/unexpected_file block_upload_test.go:1012: PASS: Exists(string,string) block_upload_test.go:1012: PASS: Get(string,string) block_upload_test.go:1012: PASS: Get(string,string) === RUN TestMultitenantCompactor_UploadBlockFile/valid_request block_upload_test.go:1012: PASS: Exists(string,string) block_upload_test.go:1012: PASS: Get(string,string) block_upload_test.go:1012: PASS: Get(string,string) block_upload_test.go:1012: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_UploadBlockFile/multiple_sequential_uploads_of_same_file --- PASS: TestMultitenantCompactor_UploadBlockFile (0.04s) --- PASS: TestMultitenantCompactor_UploadBlockFile/without_tenant_ID (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/without_block_ID (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/invalid_block_ID (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/without_path (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/invalid_path (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/empty_file (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/attempt_block_metadata_file (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/attempt_in-flight_block_metadata_file (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/block_upload_disabled (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/complete_block_already_exists (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/failure_checking_for_complete_block (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/failure_checking_for_in-flight_meta_file (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/missing_in-flight_meta_file (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/file_upload_fails (0.01s) --- PASS: TestMultitenantCompactor_UploadBlockFile/invalid_file_size (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/unexpected_file (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/valid_request (0.01s) --- PASS: TestMultitenantCompactor_UploadBlockFile/multiple_sequential_uploads_of_same_file (0.00s) === RUN TestMultitenantCompactor_FinishBlockUpload === RUN TestMultitenantCompactor_FinishBlockUpload/without_tenant_ID === RUN TestMultitenantCompactor_FinishBlockUpload/without_block_ID === RUN TestMultitenantCompactor_FinishBlockUpload/invalid_block_ID === RUN TestMultitenantCompactor_FinishBlockUpload/block_upload_disabled === RUN TestMultitenantCompactor_FinishBlockUpload/complete_block_already_exists === RUN TestMultitenantCompactor_FinishBlockUpload/checking_for_complete_block_fails === RUN TestMultitenantCompactor_FinishBlockUpload/missing_in-flight_meta_file === RUN TestMultitenantCompactor_FinishBlockUpload/downloading_in-flight_meta_file_fails === RUN TestMultitenantCompactor_FinishBlockUpload/corrupt_in-flight_meta_file === RUN TestMultitenantCompactor_FinishBlockUpload/uploading_meta_file_fails === RUN TestMultitenantCompactor_FinishBlockUpload/too_many_concurrent_validations --- PASS: TestMultitenantCompactor_FinishBlockUpload (0.02s) --- PASS: TestMultitenantCompactor_FinishBlockUpload/without_tenant_ID (0.00s) --- PASS: TestMultitenantCompactor_FinishBlockUpload/without_block_ID (0.00s) --- PASS: TestMultitenantCompactor_FinishBlockUpload/invalid_block_ID (0.00s) --- PASS: TestMultitenantCompactor_FinishBlockUpload/block_upload_disabled (0.00s) --- PASS: TestMultitenantCompactor_FinishBlockUpload/complete_block_already_exists (0.00s) --- PASS: TestMultitenantCompactor_FinishBlockUpload/checking_for_complete_block_fails (0.00s) --- PASS: TestMultitenantCompactor_FinishBlockUpload/missing_in-flight_meta_file (0.00s) --- PASS: TestMultitenantCompactor_FinishBlockUpload/downloading_in-flight_meta_file_fails (0.00s) --- PASS: TestMultitenantCompactor_FinishBlockUpload/corrupt_in-flight_meta_file (0.00s) --- PASS: TestMultitenantCompactor_FinishBlockUpload/uploading_meta_file_fails (0.00s) --- PASS: TestMultitenantCompactor_FinishBlockUpload/too_many_concurrent_validations (0.00s) === RUN TestMultitenantCompactor_ValidateAndComplete === RUN TestMultitenantCompactor_ValidateAndComplete/validation_fails === RUN TestMultitenantCompactor_ValidateAndComplete/validation_fails,_uploading_error_fails === RUN TestMultitenantCompactor_ValidateAndComplete/uploading_meta_file_fails === RUN TestMultitenantCompactor_ValidateAndComplete/uploading_meta_file_fails,_uploading_error_fails === RUN TestMultitenantCompactor_ValidateAndComplete/removing_in-flight_meta_file_fails === RUN TestMultitenantCompactor_ValidateAndComplete/removing_validation_file_fails === RUN TestMultitenantCompactor_ValidateAndComplete/valid_request --- PASS: TestMultitenantCompactor_ValidateAndComplete (0.01s) --- PASS: TestMultitenantCompactor_ValidateAndComplete/validation_fails (0.00s) --- PASS: TestMultitenantCompactor_ValidateAndComplete/validation_fails,_uploading_error_fails (0.00s) --- PASS: TestMultitenantCompactor_ValidateAndComplete/uploading_meta_file_fails (0.00s) --- PASS: TestMultitenantCompactor_ValidateAndComplete/uploading_meta_file_fails,_uploading_error_fails (0.00s) --- PASS: TestMultitenantCompactor_ValidateAndComplete/removing_in-flight_meta_file_fails (0.00s) --- PASS: TestMultitenantCompactor_ValidateAndComplete/removing_validation_file_fails (0.00s) --- PASS: TestMultitenantCompactor_ValidateAndComplete/valid_request (0.00s) === RUN TestMultitenantCompactor_ValidateBlock === RUN TestMultitenantCompactor_ValidateBlock/valid_block === RUN TestMultitenantCompactor_ValidateBlock/missing_meta_file === RUN TestMultitenantCompactor_ValidateBlock/missing_index_file === RUN TestMultitenantCompactor_ValidateBlock/missing_chunks_file === RUN TestMultitenantCompactor_ValidateBlock/file_size_mismatch === RUN TestMultitenantCompactor_ValidateBlock/empty_index_file === RUN TestMultitenantCompactor_ValidateBlock/index_file_invalid_magic_number === RUN TestMultitenantCompactor_ValidateBlock/out_of_order_labels === RUN TestMultitenantCompactor_ValidateBlock/segment_file_invalid_magic_number === RUN TestMultitenantCompactor_ValidateBlock/segment_file_invalid_checksum === RUN TestMultitenantCompactor_ValidateBlock/empty_segment_file --- PASS: TestMultitenantCompactor_ValidateBlock (15.85s) --- PASS: TestMultitenantCompactor_ValidateBlock/valid_block (1.33s) --- PASS: TestMultitenantCompactor_ValidateBlock/missing_meta_file (1.26s) --- PASS: TestMultitenantCompactor_ValidateBlock/missing_index_file (1.46s) --- PASS: TestMultitenantCompactor_ValidateBlock/missing_chunks_file (1.48s) --- PASS: TestMultitenantCompactor_ValidateBlock/file_size_mismatch (1.47s) --- PASS: TestMultitenantCompactor_ValidateBlock/empty_index_file (1.43s) --- PASS: TestMultitenantCompactor_ValidateBlock/index_file_invalid_magic_number (1.53s) --- PASS: TestMultitenantCompactor_ValidateBlock/out_of_order_labels (1.47s) --- PASS: TestMultitenantCompactor_ValidateBlock/segment_file_invalid_magic_number (1.46s) --- PASS: TestMultitenantCompactor_ValidateBlock/segment_file_invalid_checksum (1.43s) --- PASS: TestMultitenantCompactor_ValidateBlock/empty_segment_file (1.50s) === RUN TestMultitenantCompactor_PeriodicValidationUpdater === RUN TestMultitenantCompactor_PeriodicValidationUpdater/updating_validation_file_fails === RUN TestMultitenantCompactor_PeriodicValidationUpdater/updating_validation_file_succeeds === RUN TestMultitenantCompactor_PeriodicValidationUpdater/context_cancelled_before_update --- PASS: TestMultitenantCompactor_PeriodicValidationUpdater (0.10s) --- PASS: TestMultitenantCompactor_PeriodicValidationUpdater/updating_validation_file_fails (0.05s) --- PASS: TestMultitenantCompactor_PeriodicValidationUpdater/updating_validation_file_succeeds (0.05s) --- PASS: TestMultitenantCompactor_PeriodicValidationUpdater/context_cancelled_before_update (0.00s) === RUN TestMultitenantCompactor_GetBlockUploadStateHandler === RUN TestMultitenantCompactor_GetBlockUploadStateHandler/block_doesn't_exist === RUN TestMultitenantCompactor_GetBlockUploadStateHandler/complete_block === RUN TestMultitenantCompactor_GetBlockUploadStateHandler/upload_in_progress === RUN TestMultitenantCompactor_GetBlockUploadStateHandler/validating === RUN TestMultitenantCompactor_GetBlockUploadStateHandler/validation_failed === RUN TestMultitenantCompactor_GetBlockUploadStateHandler/stale_validation_file --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler (0.01s) --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler/block_doesn't_exist (0.00s) --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler/complete_block (0.00s) --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler/upload_in_progress (0.00s) --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler/validating (0.00s) --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler/validation_failed (0.00s) --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler/stale_validation_file (0.00s) === RUN TestBlocksCleaner === RUN TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=0s === PAUSE TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=0s === RUN TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=2h0m0s === PAUSE TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=2h0m0s === RUN TestBlocksCleaner/concurrency=2,_tenant_deletion_delay=0s === PAUSE TestBlocksCleaner/concurrency=2,_tenant_deletion_delay=0s === RUN TestBlocksCleaner/concurrency=10,_tenant_deletion_delay=0s === PAUSE TestBlocksCleaner/concurrency=10,_tenant_deletion_delay=0s === CONT TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=0s === CONT TestBlocksCleaner/concurrency=2,_tenant_deletion_delay=0s === CONT TestBlocksCleaner/concurrency=10,_tenant_deletion_delay=0s === CONT TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=2h0m0s --- PASS: TestBlocksCleaner (0.00s) --- PASS: TestBlocksCleaner/concurrency=2,_tenant_deletion_delay=0s (14.67s) --- PASS: TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=0s (15.22s) --- PASS: TestBlocksCleaner/concurrency=10,_tenant_deletion_delay=0s (15.20s) --- PASS: TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=2h0m0s (15.36s) === RUN TestBlocksCleaner_ShouldContinueOnBlockDeletionFailure --- PASS: TestBlocksCleaner_ShouldContinueOnBlockDeletionFailure (7.69s) === RUN TestBlocksCleaner_ShouldRebuildBucketIndexOnCorruptedOne --- PASS: TestBlocksCleaner_ShouldRebuildBucketIndexOnCorruptedOne (6.00s) === RUN TestBlocksCleaner_ShouldRemoveMetricsForTenantsNotBelongingAnymoreToTheShard --- PASS: TestBlocksCleaner_ShouldRemoveMetricsForTenantsNotBelongingAnymoreToTheShard (9.58s) === RUN TestBlocksCleaner_ShouldNotCleanupUserThatDoesntBelongToShardAnymore --- PASS: TestBlocksCleaner_ShouldNotCleanupUserThatDoesntBelongToShardAnymore (4.15s) === RUN TestBlocksCleaner_ListBlocksOutsideRetentionPeriod --- PASS: TestBlocksCleaner_ListBlocksOutsideRetentionPeriod (5.40s) === RUN TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 30.148213ms logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 9.005134ms logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level debug component cleaner user user-1 msg applying retention retention 9h0m0s logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 8.832164ms logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 18.130274ms logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level debug component cleaner user user-1 msg applying retention retention 7h0m0s logger.go:22: level info component cleaner user user-1 msg applied retention: marking block for deletion block 01H05BF23CQVM29RA3K3YN1F1Y maxTime 1683778180000 logger.go:22: level info component cleaner user user-1 msg block has been marked for deletion block 01H05BF23CQVM29RA3K3YN1F1Y logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 22.308743ms logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 8.7632ms logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level debug component cleaner user user-1 msg applying retention retention 7h0m0s logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 15.095868ms logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 16.78124ms logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level debug component cleaner user user-1 msg applying retention retention 7h0m0s logger.go:22: level debug component cleaner user user-1 msg deleted file file 01H05BF23CQVM29RA3K3YN1F1Y/meta.json bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod4048154732/001 logger.go:22: level debug component cleaner user user-1 msg deleted file file 01H05BF23CQVM29RA3K3YN1F1Y/chunks/000001 bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod4048154732/001 logger.go:22: level debug component cleaner user user-1 msg deleted file file 01H05BF23CQVM29RA3K3YN1F1Y/index bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod4048154732/001 logger.go:22: level debug component cleaner user user-1 msg deleted file file 01H05BF23CQVM29RA3K3YN1F1Y/tombstones bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod4048154732/001 logger.go:22: level debug component cleaner user user-1 msg deleted file file 01H05BF23CQVM29RA3K3YN1F1Y/deletion-mark.json bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod4048154732/001 logger.go:22: level info component cleaner user user-1 msg deleted block marked for deletion block 01H05BF23CQVM29RA3K3YN1F1Y logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 31.383321ms logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 16.297213ms logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level debug component cleaner user user-1 msg applying retention retention 7h0m0s logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 8.352938ms logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level debug component cleaner user user-2 msg applying retention retention 5h0m0s logger.go:22: level info component cleaner user user-2 msg applied retention: marking block for deletion block 01H05BFA0E48P83RM5V267EXFH maxTime 1683785388000 logger.go:22: level info component cleaner user user-2 msg block has been marked for deletion block 01H05BFA0E48P83RM5V267EXFH logger.go:22: level info component cleaner user user-2 msg applied retention: marking block for deletion block 01H05BF7JEHV0G5V58K3X9000K maxTime 1683778185000 logger.go:22: level info component cleaner user user-2 msg block has been marked for deletion block 01H05BF7JEHV0G5V58K3X9000K logger.go:22: level debug component cleaner user user-2 msg deleted file file 01H05BF7JEHV0G5V58K3X9000K/meta.json bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod4048154732/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01H05BF7JEHV0G5V58K3X9000K/chunks/000001 bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod4048154732/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01H05BF7JEHV0G5V58K3X9000K/index bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod4048154732/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01H05BF7JEHV0G5V58K3X9000K/tombstones bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod4048154732/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01H05BF7JEHV0G5V58K3X9000K/deletion-mark.json bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod4048154732/001 logger.go:22: level info component cleaner user user-2 msg deleted block marked for deletion block 01H05BF7JEHV0G5V58K3X9000K logger.go:22: level debug component cleaner user user-2 msg deleted file file 01H05BFA0E48P83RM5V267EXFH/meta.json bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod4048154732/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01H05BFA0E48P83RM5V267EXFH/chunks/000001 bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod4048154732/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01H05BFA0E48P83RM5V267EXFH/index bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod4048154732/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01H05BFA0E48P83RM5V267EXFH/tombstones bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod4048154732/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01H05BFA0E48P83RM5V267EXFH/deletion-mark.json bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod4048154732/001 logger.go:22: level info component cleaner user user-2 msg deleted block marked for deletion block 01H05BFA0E48P83RM5V267EXFH logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 32.922605ms --- PASS: TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod (10.77s) === RUN TestBlocksCleaner_ShouldRemovePartialBlocksOutsideDelayPeriod logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level warn component cleaner user user-1 msg skipped partial block when updating bucket index block 01H05BFCK4QC3R5GRFMF0J7Q3E logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 9.098419ms logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level warn component cleaner user user-1 msg skipped partial block when updating bucket index block 01H05BFCK4QC3R5GRFMF0J7Q3E logger.go:22: level info component cleaner user user-1 msg stale partial block found: marking block for deletion block 01H05BFCK4QC3R5GRFMF0J7Q3E last modified 2023-05-11 20:09:52.855154299 +0800 CST logger.go:22: level info component cleaner user user-1 msg block has been marked for deletion block 01H05BFCK4QC3R5GRFMF0J7Q3E logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 11.146692ms --- PASS: TestBlocksCleaner_ShouldRemovePartialBlocksOutsideDelayPeriod (4.04s) === RUN TestBlocksCleaner_ShouldNotRemovePartialBlocksInsideDelayPeriod logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level warn component cleaner user user-1 msg skipped partial block when updating bucket index block 01H05BFGMPFSDWDXYRY93DDHWZ logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 9.602446ms logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level error component cleaner user user-2 msg skipped block with corrupted meta.json when updating bucket index block 01H05BFJKPQ9288ME9BKQBQ2VV err unmarshal block meta file 01H05BFJKPQ9288ME9BKQBQ2VV/meta.json: invalid character 'c' looking for beginning of value: meta.json corrupted logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 7.987318ms --- PASS: TestBlocksCleaner_ShouldNotRemovePartialBlocksInsideDelayPeriod (3.80s) === RUN TestBlocksCleaner_ShouldNotRemovePartialBlocksIfConfiguredDelayIsInvalid --- PASS: TestBlocksCleaner_ShouldNotRemovePartialBlocksIfConfiguredDelayIsInvalid (1.52s) === RUN TestStalePartialBlockLastModifiedTime === RUN TestStalePartialBlockLastModifiedTime/no_objects === RUN TestStalePartialBlockLastModifiedTime/objects_newer_than_delay_cutoff === RUN TestStalePartialBlockLastModifiedTime/objects_equal_to_delay_cutoff === RUN TestStalePartialBlockLastModifiedTime/objects_older_than_delay_cutoff --- PASS: TestStalePartialBlockLastModifiedTime (1.54s) --- PASS: TestStalePartialBlockLastModifiedTime/no_objects (0.00s) --- PASS: TestStalePartialBlockLastModifiedTime/objects_newer_than_delay_cutoff (0.00s) --- PASS: TestStalePartialBlockLastModifiedTime/objects_equal_to_delay_cutoff (0.00s) --- PASS: TestStalePartialBlockLastModifiedTime/objects_older_than_delay_cutoff (0.00s) === RUN TestSyncer_GarbageCollect_e2e === PAUSE TestSyncer_GarbageCollect_e2e === RUN TestGroupCompactE2E === PAUSE TestGroupCompactE2E === RUN TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly === PAUSE TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly === RUN TestGroupKey === RUN TestGroupKey/#00 === RUN TestGroupKey/#01 === RUN TestGroupKey/#02 === RUN TestGroupKey/#03 --- PASS: TestGroupKey (0.00s) --- PASS: TestGroupKey/#00 (0.00s) --- PASS: TestGroupKey/#01 (0.00s) --- PASS: TestGroupKey/#02 (0.00s) --- PASS: TestGroupKey/#03 (0.00s) === RUN TestGroupMaxMinTime --- PASS: TestGroupMaxMinTime (0.00s) === RUN TestBucketCompactor_FilterOwnJobs === RUN TestBucketCompactor_FilterOwnJobs/should_return_all_planned_jobs_if_the_compactor_instance_owns_all_of_them === RUN TestBucketCompactor_FilterOwnJobs/should_return_no_jobs_if_the_compactor_instance_owns_none_of_them === RUN TestBucketCompactor_FilterOwnJobs/should_return_some_jobs_if_the_compactor_instance_owns_some_of_them --- PASS: TestBucketCompactor_FilterOwnJobs (0.00s) --- PASS: TestBucketCompactor_FilterOwnJobs/should_return_all_planned_jobs_if_the_compactor_instance_owns_all_of_them (0.00s) --- PASS: TestBucketCompactor_FilterOwnJobs/should_return_no_jobs_if_the_compactor_instance_owns_none_of_them (0.00s) --- PASS: TestBucketCompactor_FilterOwnJobs/should_return_some_jobs_if_the_compactor_instance_owns_some_of_them (0.00s) === RUN TestBlockMaxTimeDeltas --- PASS: TestBlockMaxTimeDeltas (0.00s) === RUN TestNoCompactionMarkFilter === RUN TestNoCompactionMarkFilter/filter_with_no_deletion_of_blocks_marked_for_no-compaction === RUN TestNoCompactionMarkFilter/filter_with_deletion_enabled === RUN TestNoCompactionMarkFilter/filter_with_deletion_enabled,_but_canceled_context === RUN TestNoCompactionMarkFilter/filtering_block_with_wrong_marker_version --- PASS: TestNoCompactionMarkFilter (0.01s) --- PASS: TestNoCompactionMarkFilter/filter_with_no_deletion_of_blocks_marked_for_no-compaction (0.00s) --- PASS: TestNoCompactionMarkFilter/filter_with_deletion_enabled (0.00s) --- PASS: TestNoCompactionMarkFilter/filter_with_deletion_enabled,_but_canceled_context (0.00s) --- PASS: TestNoCompactionMarkFilter/filtering_block_with_wrong_marker_version (0.00s) === RUN TestConvertCompactionResultToForEachJobs --- PASS: TestConvertCompactionResultToForEachJobs (0.00s) === RUN TestConfig_ShouldSupportYamlConfig --- PASS: TestConfig_ShouldSupportYamlConfig (0.00s) === RUN TestConfig_ShouldSupportCliFlags --- PASS: TestConfig_ShouldSupportCliFlags (0.00s) === RUN TestConfig_Validate === RUN TestConfig_Validate/should_pass_with_the_default_config === RUN TestConfig_Validate/should_pass_with_only_1_block_range_period === RUN TestConfig_Validate/should_fail_with_non_divisible_block_range_periods === RUN TestConfig_Validate/should_fail_on_unknown_compaction_jobs_order === RUN TestConfig_Validate/should_fail_on_invalid_value_of_max-opening-blocks-concurrency === RUN TestConfig_Validate/should_fail_on_invalid_value_of_max-closing-blocks-concurrency === RUN TestConfig_Validate/should_fail_on_invalid_value_of_symbols-flushers-concurrency --- PASS: TestConfig_Validate (0.02s) --- PASS: TestConfig_Validate/should_pass_with_the_default_config (0.00s) --- PASS: TestConfig_Validate/should_pass_with_only_1_block_range_period (0.00s) --- PASS: TestConfig_Validate/should_fail_with_non_divisible_block_range_periods (0.00s) --- PASS: TestConfig_Validate/should_fail_on_unknown_compaction_jobs_order (0.00s) --- PASS: TestConfig_Validate/should_fail_on_invalid_value_of_max-opening-blocks-concurrency (0.00s) --- PASS: TestConfig_Validate/should_fail_on_invalid_value_of_max-closing-blocks-concurrency (0.00s) --- PASS: TestConfig_Validate/should_fail_on_invalid_value_of_symbols-flushers-concurrency (0.00s) === RUN TestMultitenantCompactor_ShouldDoNothingOnNoUserBlocks === PAUSE TestMultitenantCompactor_ShouldDoNothingOnNoUserBlocks === RUN TestMultitenantCompactor_ShouldRetryCompactionOnFailureWhileDiscoveringUsersFromBucket === PAUSE TestMultitenantCompactor_ShouldRetryCompactionOnFailureWhileDiscoveringUsersFromBucket === RUN TestMultitenantCompactor_ShouldIncrementCompactionErrorIfFailedToCompactASingleTenant === PAUSE TestMultitenantCompactor_ShouldIncrementCompactionErrorIfFailedToCompactASingleTenant === RUN TestMultitenantCompactor_ShouldIncrementCompactionShutdownIfTheContextIsCancelled === PAUSE TestMultitenantCompactor_ShouldIncrementCompactionShutdownIfTheContextIsCancelled === RUN TestMultitenantCompactor_ShouldIterateOverUsersAndRunCompaction === PAUSE TestMultitenantCompactor_ShouldIterateOverUsersAndRunCompaction === RUN TestMultitenantCompactor_ShouldStopCompactingTenantOnReachingMaxCompactionTime === PAUSE TestMultitenantCompactor_ShouldStopCompactingTenantOnReachingMaxCompactionTime === RUN TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForDeletion === PAUSE TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForDeletion === RUN TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForNoCompaction === PAUSE TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForNoCompaction === RUN TestMultitenantCompactor_ShouldNotCompactBlocksForUsersMarkedForDeletion === PAUSE TestMultitenantCompactor_ShouldNotCompactBlocksForUsersMarkedForDeletion === RUN TestMultitenantCompactor_ShouldCompactAllUsersOnShardingEnabledButOnlyOneInstanceRunning === PAUSE TestMultitenantCompactor_ShouldCompactAllUsersOnShardingEnabledButOnlyOneInstanceRunning === RUN TestMultitenantCompactor_ShouldCompactOnlyUsersOwnedByTheInstanceOnShardingEnabledAndMultipleInstancesRunning === PAUSE TestMultitenantCompactor_ShouldCompactOnlyUsersOwnedByTheInstanceOnShardingEnabledAndMultipleInstancesRunning === RUN TestMultitenantCompactor_ShouldSkipCompactionForJobsNoMoreOwnedAfterPlanning === PAUSE TestMultitenantCompactor_ShouldSkipCompactionForJobsNoMoreOwnedAfterPlanning === RUN TestMultitenantCompactor_ShouldSkipCompactionForJobsWithFirstLevelCompactionBlocksAndWaitPeriodNotElapsed === PAUSE TestMultitenantCompactor_ShouldSkipCompactionForJobsWithFirstLevelCompactionBlocksAndWaitPeriodNotElapsed === RUN TestMultitenantCompactor_DeleteLocalSyncFiles --- PASS: TestMultitenantCompactor_DeleteLocalSyncFiles (6.34s) === RUN TestMultitenantCompactor_ShouldFailCompactionOnTimeout === PAUSE TestMultitenantCompactor_ShouldFailCompactionOnTimeout === RUN TestOwnUser === RUN TestOwnUser/5_compactors,_sharding_enabled,_no_compactor_shard_size === PAUSE TestOwnUser/5_compactors,_sharding_enabled,_no_compactor_shard_size === RUN TestOwnUser/10_compactors,_sharding_enabled,_with_non-zero_shard_sizes === PAUSE TestOwnUser/10_compactors,_sharding_enabled,_with_non-zero_shard_sizes === RUN TestOwnUser/10_compactors,_sharding_enabled,_with_zero_shard_size === PAUSE TestOwnUser/10_compactors,_sharding_enabled,_with_zero_shard_size === CONT TestOwnUser/5_compactors,_sharding_enabled,_no_compactor_shard_size === CONT TestOwnUser/10_compactors,_sharding_enabled,_with_non-zero_shard_sizes === CONT TestOwnUser/10_compactors,_sharding_enabled,_with_zero_shard_size --- PASS: TestOwnUser (0.00s) --- PASS: TestOwnUser/5_compactors,_sharding_enabled,_no_compactor_shard_size (0.69s) --- PASS: TestOwnUser/10_compactors,_sharding_enabled,_with_non-zero_shard_sizes (5.47s) --- PASS: TestOwnUser/10_compactors,_sharding_enabled,_with_zero_shard_size (5.83s) === RUN TestMultitenantCompactor_OutOfOrderCompaction --- PASS: TestMultitenantCompactor_OutOfOrderCompaction (2.73s) === RUN TestSortJobsBySmallestRangeOldestBlocksFirst === RUN TestSortJobsBySmallestRangeOldestBlocksFirst/should_do_nothing_on_empty_input === RUN TestSortJobsBySmallestRangeOldestBlocksFirst/should_sort_jobs_by_smallest_range,_oldest_blocks_first === RUN TestSortJobsBySmallestRangeOldestBlocksFirst/split_jobs_are_always_sorted_first --- PASS: TestSortJobsBySmallestRangeOldestBlocksFirst (0.00s) --- PASS: TestSortJobsBySmallestRangeOldestBlocksFirst/should_do_nothing_on_empty_input (0.00s) --- PASS: TestSortJobsBySmallestRangeOldestBlocksFirst/should_sort_jobs_by_smallest_range,_oldest_blocks_first (0.00s) --- PASS: TestSortJobsBySmallestRangeOldestBlocksFirst/split_jobs_are_always_sorted_first (0.00s) === RUN TestSortJobsByNewestBlocksFirst === RUN TestSortJobsByNewestBlocksFirst/should_do_nothing_on_empty_input job_sorting_test.go:127: sorted jobs: === RUN TestSortJobsByNewestBlocksFirst/should_sort_jobs_by_newest_blocks_first job_sorting_test.go:127: sorted jobs: job_sorting_test.go:129: - (minTime: 40 maxTime: 80) job_sorting_test.go:129: - (minTime: 10 maxTime: 30) job_sorting_test.go:129: - (minTime: 10 maxTime: 20) === RUN TestSortJobsByNewestBlocksFirst/should_give_precedence_to_smaller_time_ranges_in_case_of_multiple_jobs_with_the_same_max_time job_sorting_test.go:127: sorted jobs: job_sorting_test.go:129: - (minTime: 30 maxTime: 40) job_sorting_test.go:129: - (minTime: 10 maxTime: 40) === RUN TestSortJobsByNewestBlocksFirst/should_give_precedence_to_newest_blocks_over_smaller_time_ranges job_sorting_test.go:127: sorted jobs: job_sorting_test.go:129: - (minTime: 10 maxTime: 40) job_sorting_test.go:129: - (minTime: 10 maxTime: 30) job_sorting_test.go:129: - (minTime: 10 maxTime: 20) --- PASS: TestSortJobsByNewestBlocksFirst (0.00s) --- PASS: TestSortJobsByNewestBlocksFirst/should_do_nothing_on_empty_input (0.00s) --- PASS: TestSortJobsByNewestBlocksFirst/should_sort_jobs_by_newest_blocks_first (0.00s) --- PASS: TestSortJobsByNewestBlocksFirst/should_give_precedence_to_smaller_time_ranges_in_case_of_multiple_jobs_with_the_same_max_time (0.00s) --- PASS: TestSortJobsByNewestBlocksFirst/should_give_precedence_to_newest_blocks_over_smaller_time_ranges (0.00s) === RUN TestJob_MinCompactionLevel --- PASS: TestJob_MinCompactionLevel (0.00s) === RUN TestJobWaitPeriodElapsed === RUN TestJobWaitPeriodElapsed/wait_period_disabled === RUN TestJobWaitPeriodElapsed/blocks_uploaded_since_more_than_the_wait_period === RUN TestJobWaitPeriodElapsed/blocks_uploaded_since_less_than_the_wait_period === RUN TestJobWaitPeriodElapsed/blocks_uploaded_since_less_than_the_wait_period_but_their_compaction_level_is_>_1 === RUN TestJobWaitPeriodElapsed/an_error_occurred_while_checking_the_blocks_upload_timestamp --- PASS: TestJobWaitPeriodElapsed (0.02s) --- PASS: TestJobWaitPeriodElapsed/wait_period_disabled (0.00s) --- PASS: TestJobWaitPeriodElapsed/blocks_uploaded_since_more_than_the_wait_period (0.00s) --- PASS: TestJobWaitPeriodElapsed/blocks_uploaded_since_less_than_the_wait_period (0.00s) --- PASS: TestJobWaitPeriodElapsed/blocks_uploaded_since_less_than_the_wait_period_but_their_compaction_level_is_>_1 (0.00s) --- PASS: TestJobWaitPeriodElapsed/an_error_occurred_while_checking_the_blocks_upload_timestamp (0.00s) === RUN TestLabelRemoverFilter === RUN TestLabelRemoverFilter/should_remove_configured_labels_2 === RUN TestLabelRemoverFilter/should_remove_configured_labels --- PASS: TestLabelRemoverFilter (0.00s) --- PASS: TestLabelRemoverFilter/should_remove_configured_labels_2 (0.00s) --- PASS: TestLabelRemoverFilter/should_remove_configured_labels (0.00s) === RUN TestShardAwareDeduplicateFilter_Filter === RUN TestShardAwareDeduplicateFilter_Filter/two_blocks_merged_and_split_into_two,_one_shard_missing === RUN TestShardAwareDeduplicateFilter_Filter/four_base_blocks_merged_and_split_into_2_separate_shards === RUN TestShardAwareDeduplicateFilter_Filter/two_compacted_blocks_with_same_sources === RUN TestShardAwareDeduplicateFilter_Filter/compacted_blocks_with_overlapping_sources === RUN TestShardAwareDeduplicateFilter_Filter/compacted_blocks_of_level_3_with_overlapping_sources_of_different_length === RUN TestShardAwareDeduplicateFilter_Filter/blocks_with_same_sources_and_different_resolutions === RUN TestShardAwareDeduplicateFilter_Filter/block_with_invalid_shardID_cannot_'include'_its_source_blocks === RUN TestShardAwareDeduplicateFilter_Filter/when_invalid_shard_IDs_present,_no_deduplication_happens_for_source_blocks === RUN TestShardAwareDeduplicateFilter_Filter/two_blocks_merged_and_split,_with_two_shards === RUN TestShardAwareDeduplicateFilter_Filter/four_base_blocks_merged_and_split_into_2_separate_shards,_plus_another_level === RUN TestShardAwareDeduplicateFilter_Filter/compacted_block_without_sources_in_bucket === RUN TestShardAwareDeduplicateFilter_Filter/3_compacted_blocks_of_level_2_and_one_compacted_block_of_level_3_in_bucket === RUN TestShardAwareDeduplicateFilter_Filter/compacted_blocks_of_level_3_with_overlapping_sources_of_equal_length === RUN TestShardAwareDeduplicateFilter_Filter/two_blocks_merged_and_split,_with_single_shard === RUN TestShardAwareDeduplicateFilter_Filter/3_non_compacted_blocks_in_bucket === RUN TestShardAwareDeduplicateFilter_Filter/4_non_compacted_blocks_and_compacted_block_of_level_2_in_bucket === RUN TestShardAwareDeduplicateFilter_Filter/compacted_blocks_with_overlapping_sources_and_different_resolutions === RUN TestShardAwareDeduplicateFilter_Filter/two_compacted_blocks_with_overlapping_sources === RUN TestShardAwareDeduplicateFilter_Filter/compacted_blocks_of_level_3_with_overlapping_sources_of_different_length_and_different_resolutions === RUN TestShardAwareDeduplicateFilter_Filter/four_base_blocks_merged_and_split_into_2_separate_shards,_plus_another_level,_with_various_resolutions --- PASS: TestShardAwareDeduplicateFilter_Filter (0.02s) --- PASS: TestShardAwareDeduplicateFilter_Filter/two_blocks_merged_and_split_into_two,_one_shard_missing (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/four_base_blocks_merged_and_split_into_2_separate_shards (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/two_compacted_blocks_with_same_sources (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/compacted_blocks_with_overlapping_sources (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/compacted_blocks_of_level_3_with_overlapping_sources_of_different_length (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/blocks_with_same_sources_and_different_resolutions (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/block_with_invalid_shardID_cannot_'include'_its_source_blocks (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/when_invalid_shard_IDs_present,_no_deduplication_happens_for_source_blocks (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/two_blocks_merged_and_split,_with_two_shards (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/four_base_blocks_merged_and_split_into_2_separate_shards,_plus_another_level (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/compacted_block_without_sources_in_bucket (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/3_compacted_blocks_of_level_2_and_one_compacted_block_of_level_3_in_bucket (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/compacted_blocks_of_level_3_with_overlapping_sources_of_equal_length (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/two_blocks_merged_and_split,_with_single_shard (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/3_non_compacted_blocks_in_bucket (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/4_non_compacted_blocks_and_compacted_block_of_level_2_in_bucket (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/compacted_blocks_with_overlapping_sources_and_different_resolutions (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/two_compacted_blocks_with_overlapping_sources (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/compacted_blocks_of_level_3_with_overlapping_sources_of_different_length_and_different_resolutions (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/four_base_blocks_merged_and_split_into_2_separate_shards,_plus_another_level,_with_various_resolutions (0.00s) === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=root1 ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=cleaner run_id=1683807023 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=8.098684ms duration_ms=8 cached=3 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job="stage: split, range start: 0, range end: 7200000, shard: 1_of_1, blocks: 01H05BG6GPFFQFSPB76R8ZHGS5 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01H05BG7W9ZRQ7H4BANX43XV4R (min time: 1970-01-01 00:01:00 +0000 UTC, max time: 1970-01-01 00:07:00 +0000 UTC),01H05BG9SAZBAK2DQS5B3GPHFN (min time: 1970-01-01 01:00:00 +0000 UTC, max time: 1970-01-01 02:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=3 plan="[01H05BG6GPFFQFSPB76R8ZHGS5 (min time: 0, max time: 300000) 01H05BG7W9ZRQ7H4BANX43XV4R (min time: 60000, max time: 420000) 01H05BG9SAZBAK2DQS5B3GPHFN (min time: 3600000, max time: 7200000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BG6GPFFQFSPB76R8ZHGS5/meta.json level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=20.100302ms level=info component=cleaner run_id=1683807023 msg="successfully completed blocks cleanup and maintenance" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BG7W9ZRQ7H4BANX43XV4R/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BG9SAZBAK2DQS5B3GPHFN/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=3 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together2307368908/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BG6GPFFQFSPB76R8ZHGS5 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together2307368908/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BG7W9ZRQ7H4BANX43XV4R /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together2307368908/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BG9SAZBAK2DQS5B3GPHFN]" duration=14.896017ms duration_ms=14 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=3 mint=0 maxt=7200000 ulid=01H05BGBYW5AC7JQCTPPXPQHXC sources="[01H05BG7W9ZRQ7H4BANX43XV4R 01H05BG9SAZBAK2DQS5B3GPHFN 01H05BG6GPFFQFSPB76R8ZHGS5]" duration=4.108304125s shard=1_of_2 level=info component=compactor msg="compact blocks" count=3 mint=0 maxt=7200000 ulid=01H05BGBYWYFGP4TTCY0K1ZJK1 sources="[01H05BG7W9ZRQ7H4BANX43XV4R 01H05BG9SAZBAK2DQS5B3GPHFN 01H05BG6GPFFQFSPB76R8ZHGS5]" duration=4.10858334s shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compacted blocks" new="[01H05BGBYW5AC7JQCTPPXPQHXC 01H05BGBYWYFGP4TTCY0K1ZJK1]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together2307368908/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BG6GPFFQFSPB76R8ZHGS5 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together2307368908/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BG7W9ZRQ7H4BANX43XV4R /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together2307368908/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BG9SAZBAK2DQS5B3GPHFN]" duration=4.132950596s duration_ms=4132 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together2307368908/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BGBYW5AC7JQCTPPXPQHXC/chunks/000001 dst=01H05BGBYW5AC7JQCTPPXPQHXC/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together2307368908/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together2307368908/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BGBYWYFGP4TTCY0K1ZJK1/chunks/000001 dst=01H05BGBYWYFGP4TTCY0K1ZJK1/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together2307368908/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together2307368908/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BGBYWYFGP4TTCY0K1ZJK1/index dst=01H05BGBYWYFGP4TTCY0K1ZJK1/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together2307368908/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded block" result_block=01H05BGBYWYFGP4TTCY0K1ZJK1 duration=6.716048ms duration_ms=6 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together2307368908/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BGBYW5AC7JQCTPPXPQHXC/index dst=01H05BGBYW5AC7JQCTPPXPQHXC/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together2307368908/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded block" result_block=01H05BGBYW5AC7JQCTPPXPQHXC duration=16.628912ms duration_ms=16 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=28.250669ms duration_ms=28 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BG6GPFFQFSPB76R8ZHGS5 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BG6GPFFQFSPB76R8ZHGS5 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BG7W9ZRQ7H4BANX43XV4R level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BG7W9ZRQ7H4BANX43XV4R level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BG9SAZBAK2DQS5B3GPHFN level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BG9SAZBAK2DQS5B3GPHFN level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compaction job succeeded" duration=4.188251867s duration_ms=4188 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=29.697628ms duration_ms=29 cached=5 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=13.708911ms duration_ms=13 cached=5 returned=2 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/should_correctly_handle_empty_blocks_generated_in_the_splitting_stage level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=root1 ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=cleaner run_id=1683807028 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=7.825549ms level=info component=cleaner run_id=1683807028 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=3.709963ms duration_ms=3 cached=1 returned=1 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job="stage: split, range start: 7200000, range end: 14400000, shard: 1_of_1, blocks: 01H05BGGCVTQG35WKS8B4TVRXF (min time: 1970-01-01 03:59:59.999 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=1 plan="[01H05BGGCVTQG35WKS8B4TVRXF (min time: 14399999, max time: 14400000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BGGCVTQG35WKS8B4TVRXF/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=1 plan=[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorshould_correctly_handle_empty_blocks_generated_in_the_splitting_stage3942864508/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BGGCVTQG35WKS8B4TVRXF] duration=6.578121ms duration_ms=6 level=info component=compactor msg="compact blocks" count=1 mint=14399999 maxt=14400000 ulid=01H05BGHPKH8TPHX1YGVYC4PCK sources=[01H05BGGCVTQG35WKS8B4TVRXF] duration=2.01545722s shard=1_of_2 level=info component=compactor msg="compact blocks resulted in empty block" count=1 sources=[01H05BGGCVTQG35WKS8B4TVRXF] duration=2.015849082s shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new="[01H05BGHPKH8TPHX1YGVYC4PCK 00000000000000000000000000]" blocks=[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorshould_correctly_handle_empty_blocks_generated_in_the_splitting_stage3942864508/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BGGCVTQG35WKS8B4TVRXF] duration=2.016139658s duration_ms=2016 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction produced an empty block" shard_id=2_of_2 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorshould_correctly_handle_empty_blocks_generated_in_the_splitting_stage3942864508/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BGHPKH8TPHX1YGVYC4PCK/chunks/000001 dst=01H05BGHPKH8TPHX1YGVYC4PCK/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorshould_correctly_handle_empty_blocks_generated_in_the_splitting_stage3942864508/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorshould_correctly_handle_empty_blocks_generated_in_the_splitting_stage3942864508/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BGHPKH8TPHX1YGVYC4PCK/index dst=01H05BGHPKH8TPHX1YGVYC4PCK/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorshould_correctly_handle_empty_blocks_generated_in_the_splitting_stage3942864508/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01H05BGHPKH8TPHX1YGVYC4PCK duration=4.826985ms duration_ms=4 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=7.679461ms duration_ms=7 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BGGCVTQG35WKS8B4TVRXF level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BGGCVTQG35WKS8B4TVRXF level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=2.041287757s duration_ms=2041 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=5.421097ms duration_ms=5 cached=2 returned=1 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=6.414592ms duration_ms=6 cached=2 returned=1 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/splitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0 level=info component=compactor msg="instance not found in the ring" instance=root1 ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1683807050 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=25.89742ms duration_ms=25 cached=10 returned=10 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job="stage: merge, range start: 14400000, range end: 28800000, shard: 1_of_2, blocks: 01H05BGW81CWAYNCPRBB8KFPET (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 06:00:00 +0000 UTC),01H05BGZ7J2BQFVFYK7W1J3MQZ (min time: 1970-01-01 06:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job="stage: merge, range start: 14400000, range end: 28800000, shard: 2_of_2, blocks: 01H05BGXQY330WZJP3JJT5139T (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 06:00:00 +0000 UTC),01H05BH0X35X77Z957Z61Z6M0R (min time: 1970-01-01 06:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge--28800000-57600000 job="stage: merge, range start: 28800000, range end: 57600000, shard: , blocks: 01H05BH48KF2058RS546BE58FT (min time: 1970-01-01 08:00:00 +0000 UTC, max time: 1970-01-01 10:00:00 +0000 UTC),01H05BH5V0KHNZFN4NSTHD6747 (min time: 1970-01-01 14:00:00 +0000 UTC, max time: 1970-01-01 16:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge--7200000-14400000 job="stage: merge, range start: 7200000, range end: 14400000, shard: , blocks: 01H05BGS50G1PBR8Q2M3694P6Z (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC),01H05BGTS0ANCJ2F5QS2RH621S (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01H05BGTS0ANCJ2F5QS2RH621S (min time: 7200000, max time: 14400000) 01H05BGS50G1PBR8Q2M3694P6Z (min time: 7200000, max time: 14400000)]" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=20.000617ms level=info component=cleaner run_id=1683807050 msg="successfully completed blocks cleanup and maintenance" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BGTS0ANCJ2F5QS2RH621S/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BGS50G1PBR8Q2M3694P6Z/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge--7200000-14400000/01H05BGTS0ANCJ2F5QS2RH621S /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge--7200000-14400000/01H05BGS50G1PBR8Q2M3694P6Z]" duration=13.454017ms duration_ms=13 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01H05BH77CKXZJF0JMRW8GCYS5 sources="[01H05BGS50G1PBR8Q2M3694P6Z 01H05BGTS0ANCJ2F5QS2RH621S]" duration=1.437938357s shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new=[01H05BH77CKXZJF0JMRW8GCYS5] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge--7200000-14400000/01H05BGTS0ANCJ2F5QS2RH621S /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge--7200000-14400000/01H05BGS50G1PBR8Q2M3694P6Z]" duration=1.438336459s duration_ms=1438 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge--7200000-14400000/01H05BH77CKXZJF0JMRW8GCYS5/chunks/000001" dst=01H05BH77CKXZJF0JMRW8GCYS5/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge--7200000-14400000/01H05BH77CKXZJF0JMRW8GCYS5/index" dst=01H05BH77CKXZJF0JMRW8GCYS5/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01H05BH77CKXZJF0JMRW8GCYS5 duration=4.273834ms duration_ms=4 external_labels={} level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=7.772586ms duration_ms=7 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BGTS0ANCJ2F5QS2RH621S level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BGTS0ANCJ2F5QS2RH621S level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BGS50G1PBR8Q2M3694P6Z level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BGS50G1PBR8Q2M3694P6Z level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=1.467136758s duration_ms=1467 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01H05BGW81CWAYNCPRBB8KFPET (min time: 14400000, max time: 21600000) 01H05BGZ7J2BQFVFYK7W1J3MQZ (min time: 21600000, max time: 28800000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BGZ7J2BQFVFYK7W1J3MQZ/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BGW81CWAYNCPRBB8KFPET/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01H05BGW81CWAYNCPRBB8KFPET /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01H05BGZ7J2BQFVFYK7W1J3MQZ]" duration=29.847716ms duration_ms=29 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=14400000 maxt=28800000 ulid=01H05BH8NYG5V06XX18DC5ZWKJ sources="[01H05BGZ7J2BQFVFYK7W1J3MQZ 01H05BGW81CWAYNCPRBB8KFPET]" duration=1.391182194s shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compacted blocks" new=[01H05BH8NYG5V06XX18DC5ZWKJ] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01H05BGW81CWAYNCPRBB8KFPET /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01H05BGZ7J2BQFVFYK7W1J3MQZ]" duration=1.391889272s duration_ms=1391 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01H05BH8NYG5V06XX18DC5ZWKJ/chunks/000001" dst=01H05BH8NYG5V06XX18DC5ZWKJ/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01H05BH8NYG5V06XX18DC5ZWKJ/index" dst=01H05BH8NYG5V06XX18DC5ZWKJ/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded block" result_block=01H05BH8NYG5V06XX18DC5ZWKJ duration=5.550384ms duration_ms=5 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=9.66405ms duration_ms=9 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BGW81CWAYNCPRBB8KFPET level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BGW81CWAYNCPRBB8KFPET level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BGZ7J2BQFVFYK7W1J3MQZ level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BGZ7J2BQFVFYK7W1J3MQZ level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction job succeeded" duration=1.43689734s duration_ms=1436 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01H05BGXQY330WZJP3JJT5139T (min time: 14400000, max time: 21600000) 01H05BH0X35X77Z957Z61Z6M0R (min time: 21600000, max time: 28800000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BGXQY330WZJP3JJT5139T/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BH0X35X77Z957Z61Z6M0R/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01H05BGXQY330WZJP3JJT5139T /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01H05BH0X35X77Z957Z61Z6M0R]" duration=16.196728ms duration_ms=16 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=14400000 maxt=28800000 ulid=01H05BHA2AGB47P855VTY0JTPN sources="[01H05BH0X35X77Z957Z61Z6M0R 01H05BGXQY330WZJP3JJT5139T]" duration=1.410485492s shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compacted blocks" new=[01H05BHA2AGB47P855VTY0JTPN] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01H05BGXQY330WZJP3JJT5139T /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01H05BH0X35X77Z957Z61Z6M0R]" duration=1.41118025s duration_ms=1411 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01H05BHA2AGB47P855VTY0JTPN/chunks/000001" dst=01H05BHA2AGB47P855VTY0JTPN/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01H05BHA2AGB47P855VTY0JTPN/index" dst=01H05BHA2AGB47P855VTY0JTPN/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded block" result_block=01H05BHA2AGB47P855VTY0JTPN duration=5.500302ms duration_ms=5 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=9.67477ms duration_ms=9 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BGXQY330WZJP3JJT5139T level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BGXQY330WZJP3JJT5139T level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BH0X35X77Z957Z61Z6M0R level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BH0X35X77Z957Z61Z6M0R level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction job succeeded" duration=1.443307131s duration_ms=1443 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01H05BH48KF2058RS546BE58FT (min time: 28800000, max time: 36000000) 01H05BH5V0KHNZFN4NSTHD6747 (min time: 50400000, max time: 57600000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BH48KF2058RS546BE58FT/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BH5V0KHNZFN4NSTHD6747/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge--28800000-57600000/01H05BH48KF2058RS546BE58FT /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge--28800000-57600000/01H05BH5V0KHNZFN4NSTHD6747]" duration=16.555468ms duration_ms=16 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=28800000 maxt=57600000 ulid=01H05BHBFE3VN50AYGJ0EC58S2 sources="[01H05BH5V0KHNZFN4NSTHD6747 01H05BH48KF2058RS546BE58FT]" duration=3.050978635s shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compacted blocks" new=[01H05BHBFE3VN50AYGJ0EC58S2] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge--28800000-57600000/01H05BH48KF2058RS546BE58FT /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge--28800000-57600000/01H05BH5V0KHNZFN4NSTHD6747]" duration=3.051772118s duration_ms=3051 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge--28800000-57600000/01H05BHBFE3VN50AYGJ0EC58S2/chunks/000001" dst=01H05BHBFE3VN50AYGJ0EC58S2/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge--28800000-57600000/01H05BHBFE3VN50AYGJ0EC58S2/index" dst=01H05BHBFE3VN50AYGJ0EC58S2/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded block" result_block=01H05BHBFE3VN50AYGJ0EC58S2 duration=23.344799ms duration_ms=23 external_labels={} level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=29.00943ms duration_ms=29 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BH48KF2058RS546BE58FT level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BH48KF2058RS546BE58FT level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BH5V0KHNZFN4NSTHD6747 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BH5V0KHNZFN4NSTHD6747 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compaction job succeeded" duration=3.104906071s duration_ms=3104 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=37.239961ms duration_ms=37 cached=14 returned=6 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job="stage: merge, range start: 0, range end: 28800000, shard: 1_of_2, blocks: 01H05BGKY5MYFGRFK0WGKFZH9Q (min time: 1970-01-01 00:00:00.001 +0000 UTC, max time: 1970-01-01 02:00:00 +0000 UTC),01H05BH8NYG5V06XX18DC5ZWKJ (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job="stage: merge, range start: 0, range end: 28800000, shard: 2_of_2, blocks: 01H05BGNVEBZSTTQ80J9QMYG20 (min time: 1970-01-01 00:00:00.001 +0000 UTC, max time: 1970-01-01 02:00:00 +0000 UTC),01H05BHA2AGB47P855VTY0JTPN (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01H05BGKY5MYFGRFK0WGKFZH9Q (min time: 1, max time: 7200000) 01H05BH8NYG5V06XX18DC5ZWKJ (min time: 14400000, max time: 28800000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BGKY5MYFGRFK0WGKFZH9Q/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BH8NYG5V06XX18DC5ZWKJ/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge-1_of_2-0-28800000/01H05BGKY5MYFGRFK0WGKFZH9Q /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge-1_of_2-0-28800000/01H05BH8NYG5V06XX18DC5ZWKJ]" duration=30.08501ms duration_ms=30 level=info component=compactor msg="compact blocks" count=2 mint=1 maxt=28800000 ulid=01H05BHEMBVR9ERK9AE9A8J37D sources="[01H05BGKY5MYFGRFK0WGKFZH9Q 01H05BH8NYG5V06XX18DC5ZWKJ]" duration=1.616286695s shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compacted blocks" new=[01H05BHEMBVR9ERK9AE9A8J37D] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge-1_of_2-0-28800000/01H05BGKY5MYFGRFK0WGKFZH9Q /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge-1_of_2-0-28800000/01H05BH8NYG5V06XX18DC5ZWKJ]" duration=1.616666716s duration_ms=1616 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge-1_of_2-0-28800000/01H05BHEMBVR9ERK9AE9A8J37D/chunks/000001" dst=01H05BHEMBVR9ERK9AE9A8J37D/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge-1_of_2-0-28800000/01H05BHEMBVR9ERK9AE9A8J37D/index" dst=01H05BHEMBVR9ERK9AE9A8J37D/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded block" result_block=01H05BHEMBVR9ERK9AE9A8J37D duration=4.251593ms duration_ms=4 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=7.810669ms duration_ms=7 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BGKY5MYFGRFK0WGKFZH9Q level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BGKY5MYFGRFK0WGKFZH9Q level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BH8NYG5V06XX18DC5ZWKJ level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BH8NYG5V06XX18DC5ZWKJ level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction job succeeded" duration=1.660463677s duration_ms=1660 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01H05BGNVEBZSTTQ80J9QMYG20 (min time: 1, max time: 7200000) 01H05BHA2AGB47P855VTY0JTPN (min time: 14400000, max time: 28800000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BHA2AGB47P855VTY0JTPN/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BGNVEBZSTTQ80J9QMYG20/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge-2_of_2-0-28800000/01H05BGNVEBZSTTQ80J9QMYG20 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge-2_of_2-0-28800000/01H05BHA2AGB47P855VTY0JTPN]" duration=9.687891ms duration_ms=9 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=1 maxt=28800000 ulid=01H05BHG783PE3J72BB7BEC2ZB sources="[01H05BHA2AGB47P855VTY0JTPN 01H05BGNVEBZSTTQ80J9QMYG20]" duration=1.502304766s shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compacted blocks" new=[01H05BHG783PE3J72BB7BEC2ZB] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge-2_of_2-0-28800000/01H05BGNVEBZSTTQ80J9QMYG20 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge-2_of_2-0-28800000/01H05BHA2AGB47P855VTY0JTPN]" duration=1.502864317s duration_ms=1502 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge-2_of_2-0-28800000/01H05BHG783PE3J72BB7BEC2ZB/chunks/000001" dst=01H05BHG783PE3J72BB7BEC2ZB/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/001/compact/0@17241709254077376921-merge-2_of_2-0-28800000/01H05BHG783PE3J72BB7BEC2ZB/index" dst=01H05BHG783PE3J72BB7BEC2ZB/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0929060266/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded block" result_block=01H05BHG783PE3J72BB7BEC2ZB duration=6.339708ms duration_ms=6 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=10.242321ms duration_ms=10 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BGNVEBZSTTQ80J9QMYG20 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BGNVEBZSTTQ80J9QMYG20 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BHA2AGB47P855VTY0JTPN level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BHA2AGB47P855VTY0JTPN level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction job succeeded" duration=1.528101381s duration_ms=1528 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=17.014852ms duration_ms=17 cached=16 returned=4 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=39.533688ms duration_ms=39 cached=16 returned=4 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split level=info component=compactor msg="instance not found in the ring" instance=root1 ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1683807066 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=22.064729ms duration_ms=22 cached=3 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=16.683474ms level=info component=cleaner run_id=1683807066 msg="successfully completed blocks cleanup and maintenance" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job="stage: split, range start: 0, range end: 7200000, shard: 1_of_1, blocks: 01H05BHHY60C95E2S63Y1NQEQE (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01H05BHKC3TZ34XPYPK1CB9PQT (min time: 1970-01-01 00:01:00 +0000 UTC, max time: 1970-01-01 00:07:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01H05BHHY60C95E2S63Y1NQEQE (min time: 0, max time: 300000) 01H05BHKC3TZ34XPYPK1CB9PQT (min time: 60000, max time: 420000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BHKC3TZ34XPYPK1CB9PQT/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BHHY60C95E2S63Y1NQEQE/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split4201608847/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BHHY60C95E2S63Y1NQEQE /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split4201608847/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BHKC3TZ34XPYPK1CB9PQT]" duration=6.692047ms duration_ms=6 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=420000 ulid=01H05BHP715S0J2094VY3QGRXH sources="[01H05BHHY60C95E2S63Y1NQEQE 01H05BHKC3TZ34XPYPK1CB9PQT]" duration=6.448603954s shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=420000 ulid=01H05BHP71R2KNC2ZSVTH3G46V sources="[01H05BHHY60C95E2S63Y1NQEQE 01H05BHKC3TZ34XPYPK1CB9PQT]" duration=6.449089581s shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="compacted blocks" new="[01H05BHP715S0J2094VY3QGRXH 01H05BHP71R2KNC2ZSVTH3G46V]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split4201608847/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BHHY60C95E2S63Y1NQEQE /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split4201608847/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BHKC3TZ34XPYPK1CB9PQT]" duration=6.44960801s duration_ms=6449 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split4201608847/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BHP71R2KNC2ZSVTH3G46V/chunks/000001 dst=01H05BHP71R2KNC2ZSVTH3G46V/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split4201608847/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split4201608847/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BHP71R2KNC2ZSVTH3G46V/index dst=01H05BHP71R2KNC2ZSVTH3G46V/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split4201608847/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded block" result_block=01H05BHP71R2KNC2ZSVTH3G46V duration=4.18071ms duration_ms=4 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split4201608847/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BHP715S0J2094VY3QGRXH/chunks/000001 dst=01H05BHP715S0J2094VY3QGRXH/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split4201608847/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split4201608847/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BHP715S0J2094VY3QGRXH/index dst=01H05BHP715S0J2094VY3QGRXH/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split4201608847/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded block" result_block=01H05BHP715S0J2094VY3QGRXH duration=4.051743ms duration_ms=4 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=14.465593ms duration_ms=14 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BHHY60C95E2S63Y1NQEQE level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BHHY60C95E2S63Y1NQEQE level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BHKC3TZ34XPYPK1CB9PQT level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BHKC3TZ34XPYPK1CB9PQT level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="compaction job succeeded" duration=6.479303718s duration_ms=6479 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=9.008654ms duration_ms=9 cached=5 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=11.895853ms duration_ms=11 cached=5 returned=3 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/non-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=root1 ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=cleaner run_id=1683807077 msg="started blocks cleanup and maintenance" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=10.002629ms level=info component=cleaner run_id=1683807077 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=10.20552ms duration_ms=10 cached=3 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job="stage: split, range start: 0, range end: 7200000, shard: 1_of_1, blocks: 01H05BHWP21GW37YRWZ0J1NBYF (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01H05BHY9016GVW8DH4VH74EKA (min time: 1970-01-01 00:07:00 +0000 UTC, max time: 1970-01-01 00:10:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01H05BHWP21GW37YRWZ0J1NBYF (min time: 0, max time: 300000) 01H05BHY9016GVW8DH4VH74EKA (min time: 420000, max time: 600000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BHWP21GW37YRWZ0J1NBYF/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BHY9016GVW8DH4VH74EKA/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split1371607807/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BHWP21GW37YRWZ0J1NBYF /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split1371607807/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BHY9016GVW8DH4VH74EKA]" duration=6.74901ms duration_ms=6 level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=600000 ulid=01H05BJ1FPFZXR99AKWY2ZCFPR sources="[01H05BHWP21GW37YRWZ0J1NBYF 01H05BHY9016GVW8DH4VH74EKA]" duration=3.415976406s shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=600000 ulid=01H05BJ1FP6PZZBRGFKVK31A0P sources="[01H05BHWP21GW37YRWZ0J1NBYF 01H05BHY9016GVW8DH4VH74EKA]" duration=3.416201698s shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compacted blocks" new="[01H05BJ1FPFZXR99AKWY2ZCFPR 01H05BJ1FP6PZZBRGFKVK31A0P]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split1371607807/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BHWP21GW37YRWZ0J1NBYF /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split1371607807/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BHY9016GVW8DH4VH74EKA]" duration=3.416554518s duration_ms=3416 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split1371607807/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BJ1FP6PZZBRGFKVK31A0P/chunks/000001 dst=01H05BJ1FP6PZZBRGFKVK31A0P/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split1371607807/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split1371607807/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BJ1FP6PZZBRGFKVK31A0P/index dst=01H05BJ1FP6PZZBRGFKVK31A0P/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split1371607807/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded block" result_block=01H05BJ1FP6PZZBRGFKVK31A0P duration=4.490966ms duration_ms=4 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split1371607807/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BJ1FPFZXR99AKWY2ZCFPR/chunks/000001 dst=01H05BJ1FPFZXR99AKWY2ZCFPR/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split1371607807/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split1371607807/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BJ1FPFZXR99AKWY2ZCFPR/index dst=01H05BJ1FPFZXR99AKWY2ZCFPR/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split1371607807/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded block" result_block=01H05BJ1FPFZXR99AKWY2ZCFPR duration=30.535754ms duration_ms=30 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=78.544946ms duration_ms=78 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BHWP21GW37YRWZ0J1NBYF level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BHWP21GW37YRWZ0J1NBYF level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BHY9016GVW8DH4VH74EKA level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BHY9016GVW8DH4VH74EKA level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction job succeeded" duration=3.507628951s duration_ms=3507 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=24.832241ms duration_ms=24 cached=5 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=12.916548ms duration_ms=12 cached=5 returned=3 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/non-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=root1 ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1683807086 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=10.631463ms level=info component=cleaner run_id=1683807086 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=28.26971ms duration_ms=28 cached=3 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job="stage: split, range start: 0, range end: 7200000, shard: 1_of_1, blocks: 01H05BJ5A4C11TV713NT0624ZJ (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01H05BJ6W04VG7GHDD2HFR403M (min time: 1970-01-01 00:05:00 +0000 UTC, max time: 1970-01-01 00:10:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01H05BJ5A4C11TV713NT0624ZJ (min time: 0, max time: 300000) 01H05BJ6W04VG7GHDD2HFR403M (min time: 300000, max time: 600000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BJ5A4C11TV713NT0624ZJ/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BJ6W04VG7GHDD2HFR403M/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split1762232692/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BJ5A4C11TV713NT0624ZJ /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split1762232692/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BJ6W04VG7GHDD2HFR403M]" duration=8.846405ms duration_ms=8 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=600000 ulid=01H05BJ9FNMMRSEBHZ9YBQRW9C sources="[01H05BJ6W04VG7GHDD2HFR403M 01H05BJ5A4C11TV713NT0624ZJ]" duration=2.726931588s shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=600000 ulid=01H05BJ9FN3PK31EXHJNEXPGWG sources="[01H05BJ6W04VG7GHDD2HFR403M 01H05BJ5A4C11TV713NT0624ZJ]" duration=2.7271436s shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compacted blocks" new="[01H05BJ9FNMMRSEBHZ9YBQRW9C 01H05BJ9FN3PK31EXHJNEXPGWG]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split1762232692/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BJ5A4C11TV713NT0624ZJ /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split1762232692/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BJ6W04VG7GHDD2HFR403M]" duration=2.727412255s duration_ms=2727 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split1762232692/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BJ9FNMMRSEBHZ9YBQRW9C/chunks/000001 dst=01H05BJ9FNMMRSEBHZ9YBQRW9C/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split1762232692/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split1762232692/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BJ9FN3PK31EXHJNEXPGWG/chunks/000001 dst=01H05BJ9FN3PK31EXHJNEXPGWG/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split1762232692/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split1762232692/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BJ9FNMMRSEBHZ9YBQRW9C/index dst=01H05BJ9FNMMRSEBHZ9YBQRW9C/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split1762232692/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split1762232692/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BJ9FN3PK31EXHJNEXPGWG/index dst=01H05BJ9FN3PK31EXHJNEXPGWG/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split1762232692/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded block" result_block=01H05BJ9FNMMRSEBHZ9YBQRW9C duration=3.454909ms duration_ms=3 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded block" result_block=01H05BJ9FN3PK31EXHJNEXPGWG duration=4.454804ms duration_ms=4 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=7.232877ms duration_ms=7 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BJ5A4C11TV713NT0624ZJ level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BJ5A4C11TV713NT0624ZJ level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BJ6W04VG7GHDD2HFR403M level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BJ6W04VG7GHDD2HFR403M level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction job succeeded" duration=2.749516987s duration_ms=2749 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=8.323176ms duration_ms=8 cached=5 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=9.527403ms duration_ms=9 cached=5 returned=3 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/smaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=root1 ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1683807107 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=16.849404ms duration_ms=16 cached=10 returned=10 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job="stage: merge, range start: 14400000, range end: 28800000, shard: 1_of_2, blocks: 01H05BJKXK4VSFBVBA08J8J85Y (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 06:00:00 +0000 UTC),01H05BJPTTTQSRPVCSZ52N2HAN (min time: 1970-01-01 06:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job="stage: merge, range start: 14400000, range end: 28800000, shard: 2_of_2, blocks: 01H05BJNBA79Y6MC0QBS508QMM (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 06:00:00 +0000 UTC),01H05BJRFD48ER91GB8WTP2HTK (min time: 1970-01-01 06:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 job="stage: split, range start: 28800000, range end: 36000000, shard: 1_of_1, blocks: 01H05BJT8RRD5MPBHHQTM20DJ2 (min time: 1970-01-01 08:00:00 +0000 UTC, max time: 1970-01-01 10:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 job="stage: split, range start: 50400000, range end: 57600000, shard: 1_of_1, blocks: 01H05BJVPYPPDMA187PN0VJVR6 (min time: 1970-01-01 14:00:00 +0000 UTC, max time: 1970-01-01 16:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job="stage: split, range start: 7200000, range end: 14400000, shard: 1_of_1, blocks: 01H05BJGVFXTW89DD0J4CNV2N5 (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC),01H05BJJE82FN21XWN6P86CHMS (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01H05BJGVFXTW89DD0J4CNV2N5 (min time: 7200000, max time: 14400000) 01H05BJJE82FN21XWN6P86CHMS (min time: 7200000, max time: 14400000)]" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=26.740986ms level=info component=cleaner run_id=1683807107 msg="successfully completed blocks cleanup and maintenance" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BJJE82FN21XWN6P86CHMS/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BJGVFXTW89DD0J4CNV2N5/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BJGVFXTW89DD0J4CNV2N5 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BJJE82FN21XWN6P86CHMS]" duration=28.140583ms duration_ms=28 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01H05BJYX9FTZGABYC96HZPM0R sources="[01H05BJGVFXTW89DD0J4CNV2N5 01H05BJJE82FN21XWN6P86CHMS]" duration=3.546064179s shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01H05BJYX929S8WA0CKE5MEAPV sources="[01H05BJGVFXTW89DD0J4CNV2N5 01H05BJJE82FN21XWN6P86CHMS]" duration=3.5464618s shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new="[01H05BJYX9FTZGABYC96HZPM0R 01H05BJYX929S8WA0CKE5MEAPV]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BJGVFXTW89DD0J4CNV2N5 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BJJE82FN21XWN6P86CHMS]" duration=3.546836221s duration_ms=3546 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BJYX9FTZGABYC96HZPM0R/chunks/000001 dst=01H05BJYX9FTZGABYC96HZPM0R/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BJYX9FTZGABYC96HZPM0R/index dst=01H05BJYX9FTZGABYC96HZPM0R/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BJYX929S8WA0CKE5MEAPV/chunks/000001 dst=01H05BJYX929S8WA0CKE5MEAPV/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01H05BJYX9FTZGABYC96HZPM0R duration=4.352879ms duration_ms=4 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BJYX929S8WA0CKE5MEAPV/index dst=01H05BJYX929S8WA0CKE5MEAPV/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01H05BJYX929S8WA0CKE5MEAPV duration=13.183762ms duration_ms=13 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=16.764919ms duration_ms=16 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BJGVFXTW89DD0J4CNV2N5 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BJGVFXTW89DD0J4CNV2N5 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BJJE82FN21XWN6P86CHMS level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BJJE82FN21XWN6P86CHMS level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=3.599203572s duration_ms=3599 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=1 plan="[01H05BJT8RRD5MPBHHQTM20DJ2 (min time: 28800000, max time: 36000000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BJT8RRD5MPBHHQTM20DJ2/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=1 plan=[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-split-1_of_1-28800000-36000000/01H05BJT8RRD5MPBHHQTM20DJ2] duration=8.655514ms duration_ms=8 level=info component=compactor msg="compact blocks" count=1 mint=28800000 maxt=36000000 ulid=01H05BK2D31WJ0GR49DW8ARBXT sources=[01H05BJT8RRD5MPBHHQTM20DJ2] duration=3.649593055s shard=1_of_2 level=info component=compactor msg="compact blocks" count=1 mint=28800000 maxt=36000000 ulid=01H05BK2D3CYAZ69C5RP51BRE6 sources=[01H05BJT8RRD5MPBHHQTM20DJ2] duration=3.649887951s shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="compacted blocks" new="[01H05BK2D31WJ0GR49DW8ARBXT 01H05BK2D3CYAZ69C5RP51BRE6]" blocks=[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-split-1_of_1-28800000-36000000/01H05BJT8RRD5MPBHHQTM20DJ2] duration=3.650102682s duration_ms=3650 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-split-1_of_1-28800000-36000000/01H05BK2D3CYAZ69C5RP51BRE6/chunks/000001 dst=01H05BK2D3CYAZ69C5RP51BRE6/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-split-1_of_1-28800000-36000000/01H05BK2D3CYAZ69C5RP51BRE6/index dst=01H05BK2D3CYAZ69C5RP51BRE6/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="uploaded block" result_block=01H05BK2D3CYAZ69C5RP51BRE6 duration=5.403976ms duration_ms=5 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-split-1_of_1-28800000-36000000/01H05BK2D31WJ0GR49DW8ARBXT/chunks/000001 dst=01H05BK2D31WJ0GR49DW8ARBXT/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-split-1_of_1-28800000-36000000/01H05BK2D31WJ0GR49DW8ARBXT/index dst=01H05BK2D31WJ0GR49DW8ARBXT/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="uploaded block" result_block=01H05BK2D31WJ0GR49DW8ARBXT duration=8.20781ms duration_ms=8 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=12.905988ms duration_ms=12 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BJT8RRD5MPBHHQTM20DJ2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BJT8RRD5MPBHHQTM20DJ2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="compaction job succeeded" duration=3.675800731s duration_ms=3675 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=1 plan="[01H05BJVPYPPDMA187PN0VJVR6 (min time: 50400000, max time: 57600000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BJVPYPPDMA187PN0VJVR6/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=1 plan=[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-split-1_of_1-50400000-57600000/01H05BJVPYPPDMA187PN0VJVR6] duration=39.540247ms duration_ms=39 level=info component=compactor msg="compact blocks" count=1 mint=50400000 maxt=57600000 ulid=01H05BK615XNF5NC45TJ6Q2178 sources=[01H05BJVPYPPDMA187PN0VJVR6] duration=3.552074588s shard=1_of_2 level=info component=compactor msg="compact blocks" count=1 mint=50400000 maxt=57600000 ulid=01H05BK6153VY1GJ7JEVKB7Z9Z sources=[01H05BJVPYPPDMA187PN0VJVR6] duration=3.552265799s shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compacted blocks" new="[01H05BK615XNF5NC45TJ6Q2178 01H05BK6153VY1GJ7JEVKB7Z9Z]" blocks=[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-split-1_of_1-50400000-57600000/01H05BJVPYPPDMA187PN0VJVR6] duration=3.552589816s duration_ms=3552 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-split-1_of_1-50400000-57600000/01H05BK6153VY1GJ7JEVKB7Z9Z/chunks/000001 dst=01H05BK6153VY1GJ7JEVKB7Z9Z/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-split-1_of_1-50400000-57600000/01H05BK6153VY1GJ7JEVKB7Z9Z/index dst=01H05BK6153VY1GJ7JEVKB7Z9Z/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded block" result_block=01H05BK6153VY1GJ7JEVKB7Z9Z duration=4.575611ms duration_ms=4 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-split-1_of_1-50400000-57600000/01H05BK615XNF5NC45TJ6Q2178/chunks/000001 dst=01H05BK615XNF5NC45TJ6Q2178/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-split-1_of_1-50400000-57600000/01H05BK615XNF5NC45TJ6Q2178/index dst=01H05BK615XNF5NC45TJ6Q2178/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded block" result_block=01H05BK615XNF5NC45TJ6Q2178 duration=3.937976ms duration_ms=3 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=14.262542ms duration_ms=14 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BJVPYPPDMA187PN0VJVR6 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BJVPYPPDMA187PN0VJVR6 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compaction job succeeded" duration=3.616052496s duration_ms=3616 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01H05BJKXK4VSFBVBA08J8J85Y (min time: 14400000, max time: 21600000) 01H05BJPTTTQSRPVCSZ52N2HAN (min time: 21600000, max time: 28800000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BJPTTTQSRPVCSZ52N2HAN/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BJKXK4VSFBVBA08J8J85Y/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01H05BJKXK4VSFBVBA08J8J85Y /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01H05BJPTTTQSRPVCSZ52N2HAN]" duration=13.751154ms duration_ms=13 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=14400000 maxt=28800000 ulid=01H05BK9H8ZPHGXF1RJ8ZEH0ET sources="[01H05BJPTTTQSRPVCSZ52N2HAN 01H05BJKXK4VSFBVBA08J8J85Y]" duration=1.223476679s shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compacted blocks" new=[01H05BK9H8ZPHGXF1RJ8ZEH0ET] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01H05BJKXK4VSFBVBA08J8J85Y /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01H05BJPTTTQSRPVCSZ52N2HAN]" duration=1.224159116s duration_ms=1224 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01H05BK9H8ZPHGXF1RJ8ZEH0ET/chunks/000001 dst=01H05BK9H8ZPHGXF1RJ8ZEH0ET/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01H05BK9H8ZPHGXF1RJ8ZEH0ET/index dst=01H05BK9H8ZPHGXF1RJ8ZEH0ET/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded block" result_block=01H05BK9H8ZPHGXF1RJ8ZEH0ET duration=4.898989ms duration_ms=4 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=8.959691ms duration_ms=8 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BJKXK4VSFBVBA08J8J85Y level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BJKXK4VSFBVBA08J8J85Y level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BJPTTTQSRPVCSZ52N2HAN level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BJPTTTQSRPVCSZ52N2HAN level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction job succeeded" duration=1.25231746s duration_ms=1252 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01H05BJNBA79Y6MC0QBS508QMM (min time: 14400000, max time: 21600000) 01H05BJRFD48ER91GB8WTP2HTK (min time: 21600000, max time: 28800000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BJRFD48ER91GB8WTP2HTK/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BJNBA79Y6MC0QBS508QMM/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01H05BJNBA79Y6MC0QBS508QMM /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01H05BJRFD48ER91GB8WTP2HTK]" duration=15.861509ms duration_ms=15 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=14400000 maxt=28800000 ulid=01H05BKARF25VE50S7RPDRQG8D sources="[01H05BJRFD48ER91GB8WTP2HTK 01H05BJNBA79Y6MC0QBS508QMM]" duration=1.21053581s shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compacted blocks" new=[01H05BKARF25VE50S7RPDRQG8D] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01H05BJNBA79Y6MC0QBS508QMM /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01H05BJRFD48ER91GB8WTP2HTK]" duration=1.21109088s duration_ms=1211 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01H05BKARF25VE50S7RPDRQG8D/chunks/000001 dst=01H05BKARF25VE50S7RPDRQG8D/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01H05BKARF25VE50S7RPDRQG8D/index dst=01H05BKARF25VE50S7RPDRQG8D/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded block" result_block=01H05BKARF25VE50S7RPDRQG8D duration=5.267809ms duration_ms=5 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=8.847525ms duration_ms=8 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BJNBA79Y6MC0QBS508QMM level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BJNBA79Y6MC0QBS508QMM level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BJRFD48ER91GB8WTP2HTK level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BJRFD48ER91GB8WTP2HTK level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction job succeeded" duration=1.241689358s duration_ms=1241 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=19.358981ms duration_ms=19 cached=18 returned=10 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 job="stage: merge, range start: 0, range end: 14400000, shard: 1_of_2, blocks: 01H05BJCF231PRXRR7JMPQ8K56 (min time: 1970-01-01 00:00:00.001 +0000 UTC, max time: 1970-01-01 02:00:00 +0000 UTC),01H05BJYX9FTZGABYC96HZPM0R (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 job="stage: merge, range start: 0, range end: 14400000, shard: 2_of_2, blocks: 01H05BJDW9G57GHQV38VTA9009 (min time: 1970-01-01 00:00:00.001 +0000 UTC, max time: 1970-01-01 02:00:00 +0000 UTC),01H05BJYX929S8WA0CKE5MEAPV (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 job="stage: merge, range start: 28800000, range end: 57600000, shard: 1_of_2, blocks: 01H05BK2D31WJ0GR49DW8ARBXT (min time: 1970-01-01 08:00:00 +0000 UTC, max time: 1970-01-01 10:00:00 +0000 UTC),01H05BK615XNF5NC45TJ6Q2178 (min time: 1970-01-01 14:00:00 +0000 UTC, max time: 1970-01-01 16:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 job="stage: merge, range start: 28800000, range end: 57600000, shard: 2_of_2, blocks: 01H05BK2D3CYAZ69C5RP51BRE6 (min time: 1970-01-01 08:00:00 +0000 UTC, max time: 1970-01-01 10:00:00 +0000 UTC),01H05BK6153VY1GJ7JEVKB7Z9Z (min time: 1970-01-01 14:00:00 +0000 UTC, max time: 1970-01-01 16:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01H05BJCF231PRXRR7JMPQ8K56 (min time: 1, max time: 7200000) 01H05BJYX9FTZGABYC96HZPM0R (min time: 7200000, max time: 14400000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BJCF231PRXRR7JMPQ8K56/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BJYX9FTZGABYC96HZPM0R/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-merge-1_of_2-0-14400000/01H05BJCF231PRXRR7JMPQ8K56 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-merge-1_of_2-0-14400000/01H05BJYX9FTZGABYC96HZPM0R]" duration=8.065562ms duration_ms=8 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=1 maxt=14400000 ulid=01H05BKBZSMNWQYYX6Y15J9KR6 sources="[01H05BJYX9FTZGABYC96HZPM0R 01H05BJCF231PRXRR7JMPQ8K56]" duration=1.500344819s shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new=[01H05BKBZSMNWQYYX6Y15J9KR6] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-merge-1_of_2-0-14400000/01H05BJCF231PRXRR7JMPQ8K56 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-merge-1_of_2-0-14400000/01H05BJYX9FTZGABYC96HZPM0R]" duration=1.500710918s duration_ms=1500 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-merge-1_of_2-0-14400000/01H05BKBZSMNWQYYX6Y15J9KR6/chunks/000001 dst=01H05BKBZSMNWQYYX6Y15J9KR6/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-merge-1_of_2-0-14400000/01H05BKBZSMNWQYYX6Y15J9KR6/index dst=01H05BKBZSMNWQYYX6Y15J9KR6/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01H05BKBZSMNWQYYX6Y15J9KR6 duration=4.450004ms duration_ms=4 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=8.128606ms duration_ms=8 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BJCF231PRXRR7JMPQ8K56 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BJCF231PRXRR7JMPQ8K56 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BJYX9FTZGABYC96HZPM0R level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BJYX9FTZGABYC96HZPM0R level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=1.521747592s duration_ms=1521 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01H05BJDW9G57GHQV38VTA9009 (min time: 1, max time: 7200000) 01H05BJYX929S8WA0CKE5MEAPV (min time: 7200000, max time: 14400000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BJYX929S8WA0CKE5MEAPV/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BJDW9G57GHQV38VTA9009/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-merge-2_of_2-0-14400000/01H05BJDW9G57GHQV38VTA9009 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-merge-2_of_2-0-14400000/01H05BJYX929S8WA0CKE5MEAPV]" duration=7.210316ms duration_ms=7 level=info component=compactor msg="Found overlapping blocks during compaction" split_merge_compactor_test.go:710: expected , got Diff: --- metric output does not match expectation; want +++ got: @@ -2,3 +2,3 @@ # TYPE cortex_compactor_runs_completed_total counter -cortex_compactor_runs_completed_total 1 +cortex_compactor_runs_completed_total 0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor level=error component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job failed" duration=1.194065946s duration_ms=1194 err="compact blocks [/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-merge-2_of_2-0-14400000/01H05BJDW9G57GHQV38VTA9009 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges2370952254/001/compact/0@17241709254077376921-merge-2_of_2-0-14400000/01H05BJYX929S8WA0CKE5MEAPV]: 2 errors: populate block: adding series: asyncBlockWriter doesn't run anymore: add series: context canceled; context canceled" level=info component=compactor msg="compaction for user was interrupted by a shutdown" user=user-1 === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=root1 ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=cleaner run_id=1683807129 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=18.630461ms duration_ms=18 cached=3 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job="stage: split, range start: 0, range end: 7200000, shard: 1_of_1, blocks: 01H05BKER2E4N8QP7PT7ZJX9FJ (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01H05BKG5X8C3XPG9RK35HAMMN (min time: 1970-01-01 00:01:00 +0000 UTC, max time: 1970-01-01 00:07:00 +0000 UTC),01H05BKJ2KYRKG7AT52SWTVBRB (min time: 1970-01-01 01:00:00 +0000 UTC, max time: 1970-01-01 02:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=3 plan="[01H05BKER2E4N8QP7PT7ZJX9FJ (min time: 0, max time: 300000) 01H05BKG5X8C3XPG9RK35HAMMN (min time: 60000, max time: 420000) 01H05BKJ2KYRKG7AT52SWTVBRB (min time: 3600000, max time: 7200000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BKER2E4N8QP7PT7ZJX9FJ/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BKG5X8C3XPG9RK35HAMMN/meta.json level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=23.869948ms level=info component=cleaner run_id=1683807129 msg="successfully completed blocks cleanup and maintenance" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BKJ2KYRKG7AT52SWTVBRB/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=3 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together939147481/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BKER2E4N8QP7PT7ZJX9FJ /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together939147481/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BKG5X8C3XPG9RK35HAMMN /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together939147481/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BKJ2KYRKG7AT52SWTVBRB]" duration=30.44487ms duration_ms=30 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=3 mint=0 maxt=7200000 ulid=01H05BKM9MKH9TT0F0H5V3AZ9V sources="[01H05BKG5X8C3XPG9RK35HAMMN 01H05BKJ2KYRKG7AT52SWTVBRB 01H05BKER2E4N8QP7PT7ZJX9FJ]" duration=5.604315105s shard=1_of_2 level=info component=compactor msg="compact blocks" count=3 mint=0 maxt=7200000 ulid=01H05BKM9MZ6QDYG3AYJWD7Z19 sources="[01H05BKG5X8C3XPG9RK35HAMMN 01H05BKJ2KYRKG7AT52SWTVBRB 01H05BKER2E4N8QP7PT7ZJX9FJ]" duration=5.646129077s shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compacted blocks" new="[01H05BKM9MKH9TT0F0H5V3AZ9V 01H05BKM9MZ6QDYG3AYJWD7Z19]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together939147481/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BKER2E4N8QP7PT7ZJX9FJ /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together939147481/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BKG5X8C3XPG9RK35HAMMN /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together939147481/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BKJ2KYRKG7AT52SWTVBRB]" duration=5.64708065s duration_ms=5647 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together939147481/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BKM9MKH9TT0F0H5V3AZ9V/chunks/000001 dst=01H05BKM9MKH9TT0F0H5V3AZ9V/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together939147481/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together939147481/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BKM9MKH9TT0F0H5V3AZ9V/index dst=01H05BKM9MKH9TT0F0H5V3AZ9V/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together939147481/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded block" result_block=01H05BKM9MKH9TT0F0H5V3AZ9V duration=5.326212ms duration_ms=5 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together939147481/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BKM9MZ6QDYG3AYJWD7Z19/chunks/000001 dst=01H05BKM9MZ6QDYG3AYJWD7Z19/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together939147481/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together939147481/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01H05BKM9MZ6QDYG3AYJWD7Z19/index dst=01H05BKM9MZ6QDYG3AYJWD7Z19/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together939147481/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded block" result_block=01H05BKM9MZ6QDYG3AYJWD7Z19 duration=4.962352ms duration_ms=4 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=19.425865ms duration_ms=19 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BKER2E4N8QP7PT7ZJX9FJ level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BKER2E4N8QP7PT7ZJX9FJ level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BKG5X8C3XPG9RK35HAMMN level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BKG5X8C3XPG9RK35HAMMN level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BKJ2KYRKG7AT52SWTVBRB level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BKJ2KYRKG7AT52SWTVBRB level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compaction job succeeded" duration=5.706477827s duration_ms=5706 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=29.219042ms duration_ms=29 cached=5 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=28.656131ms duration_ms=28 cached=5 returned=2 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/splitting_should_be_disabled_if_configured_shards_=_0 level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=root1 ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1683807143 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=11.590395ms level=info component=cleaner run_id=1683807143 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=20.05982ms duration_ms=20 cached=3 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge--0-7200000 job="stage: merge, range start: 0, range end: 7200000, shard: , blocks: 01H05BKT8061GEVKH464D7VY18 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01H05BKX1XYJDDFNK819340SS5 (min time: 1970-01-01 00:05:00 +0000 UTC, max time: 1970-01-01 00:10:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01H05BKT8061GEVKH464D7VY18 (min time: 0, max time: 300000) 01H05BKX1XYJDDFNK819340SS5 (min time: 300000, max time: 600000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BKX1XYJDDFNK819340SS5/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BKT8061GEVKH464D7VY18/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_02430767860/001/compact/0@17241709254077376921-merge--0-7200000/01H05BKT8061GEVKH464D7VY18 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_02430767860/001/compact/0@17241709254077376921-merge--0-7200000/01H05BKX1XYJDDFNK819340SS5]" duration=15.078427ms duration_ms=15 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=600000 ulid=01H05BM1BRVQ2VJBD6XREXF3EH sources="[01H05BKX1XYJDDFNK819340SS5 01H05BKT8061GEVKH464D7VY18]" duration=1.94138244s shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compacted blocks" new=[01H05BM1BRVQ2VJBD6XREXF3EH] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_02430767860/001/compact/0@17241709254077376921-merge--0-7200000/01H05BKT8061GEVKH464D7VY18 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_02430767860/001/compact/0@17241709254077376921-merge--0-7200000/01H05BKX1XYJDDFNK819340SS5]" duration=1.945089523s duration_ms=1945 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_02430767860/001/compact/0@17241709254077376921-merge--0-7200000/01H05BM1BRVQ2VJBD6XREXF3EH/chunks/000001" dst=01H05BM1BRVQ2VJBD6XREXF3EH/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_02430767860/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_02430767860/001/compact/0@17241709254077376921-merge--0-7200000/01H05BM1BRVQ2VJBD6XREXF3EH/index" dst=01H05BM1BRVQ2VJBD6XREXF3EH/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_02430767860/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded block" result_block=01H05BM1BRVQ2VJBD6XREXF3EH duration=11.764645ms duration_ms=11 external_labels={} level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=23.357441ms duration_ms=23 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BKT8061GEVKH464D7VY18 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BKT8061GEVKH464D7VY18 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BKX1XYJDDFNK819340SS5 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BKX1XYJDDFNK819340SS5 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction job succeeded" duration=1.998946555s duration_ms=1998 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=5.363174ms duration_ms=5 cached=4 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=14.969461ms duration_ms=14 cached=4 returned=2 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/compaction_on_blocks_containing_native_histograms level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=root1 ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1683807149 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=30.153493ms duration_ms=30 cached=2 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job="stage: split, range start: 7200000, range end: 14400000, shard: 1_of_1, blocks: 01H05BM3J7NA18DHY88AAR3SBS (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC),01H05BM5NCH1R1BWSFYQDYN8DJ (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01H05BM3J7NA18DHY88AAR3SBS (min time: 7200000, max time: 14400000) 01H05BM5NCH1R1BWSFYQDYN8DJ (min time: 7200000, max time: 14400000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BM3J7NA18DHY88AAR3SBS/meta.json level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=38.067847ms level=info component=cleaner run_id=1683807149 msg="successfully completed blocks cleanup and maintenance" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BM5NCH1R1BWSFYQDYN8DJ/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorcompaction_on_blocks_containing_native_histograms1959001627/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BM3J7NA18DHY88AAR3SBS /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorcompaction_on_blocks_containing_native_histograms1959001627/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BM5NCH1R1BWSFYQDYN8DJ]" duration=26.162555ms duration_ms=26 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01H05BM7M62KF70XQ32S9HTCJ0 sources="[01H05BM3J7NA18DHY88AAR3SBS 01H05BM5NCH1R1BWSFYQDYN8DJ]" duration=4.851087409s shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01H05BM7M6J0NFDC5WA5YK8FTH sources="[01H05BM3J7NA18DHY88AAR3SBS 01H05BM5NCH1R1BWSFYQDYN8DJ]" duration=4.859432026s shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new="[01H05BM7M62KF70XQ32S9HTCJ0 01H05BM7M6J0NFDC5WA5YK8FTH]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorcompaction_on_blocks_containing_native_histograms1959001627/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BM3J7NA18DHY88AAR3SBS /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorcompaction_on_blocks_containing_native_histograms1959001627/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BM5NCH1R1BWSFYQDYN8DJ]" duration=4.860275752s duration_ms=4860 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorcompaction_on_blocks_containing_native_histograms1959001627/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BM7M62KF70XQ32S9HTCJ0/chunks/000001 dst=01H05BM7M62KF70XQ32S9HTCJ0/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorcompaction_on_blocks_containing_native_histograms1959001627/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorcompaction_on_blocks_containing_native_histograms1959001627/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BM7M62KF70XQ32S9HTCJ0/index dst=01H05BM7M62KF70XQ32S9HTCJ0/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorcompaction_on_blocks_containing_native_histograms1959001627/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01H05BM7M62KF70XQ32S9HTCJ0 duration=4.438643ms duration_ms=4 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorcompaction_on_blocks_containing_native_histograms1959001627/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BM7M6J0NFDC5WA5YK8FTH/chunks/000001 dst=01H05BM7M6J0NFDC5WA5YK8FTH/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorcompaction_on_blocks_containing_native_histograms1959001627/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorcompaction_on_blocks_containing_native_histograms1959001627/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BM7M6J0NFDC5WA5YK8FTH/index dst=01H05BM7M6J0NFDC5WA5YK8FTH/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorcompaction_on_blocks_containing_native_histograms1959001627/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01H05BM7M6J0NFDC5WA5YK8FTH duration=3.876852ms duration_ms=3 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=17.477598ms duration_ms=17 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BM3J7NA18DHY88AAR3SBS level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BM3J7NA18DHY88AAR3SBS level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BM5NCH1R1BWSFYQDYN8DJ level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BM5NCH1R1BWSFYQDYN8DJ level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=4.909996718s duration_ms=4909 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=9.086258ms duration_ms=9 cached=4 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=28.394677ms duration_ms=28 cached=4 returned=2 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=root1 ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=cleaner run_id=1683807158 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=8.861926ms level=info component=cleaner run_id=1683807158 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=6.226261ms duration_ms=6 cached=2 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job="stage: split, range start: 7200000, range end: 14400000, shard: 1_of_1, blocks: 01H05BMCP13JF1FE0M3ZE6PXY1 (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC),01H05BMF3DDHKRRT81YBE9PATA (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01H05BMCP13JF1FE0M3ZE6PXY1 (min time: 7200000, max time: 14400000) 01H05BMF3DDHKRRT81YBE9PATA (min time: 7200000, max time: 14400000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BMCP13JF1FE0M3ZE6PXY1/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BMF3DDHKRRT81YBE9PATA/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3102621333/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BMCP13JF1FE0M3ZE6PXY1 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3102621333/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BMF3DDHKRRT81YBE9PATA]" duration=8.288454ms duration_ms=8 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01H05BMGG5XCYY9KDZKRNERBKF sources="[01H05BMF3DDHKRRT81YBE9PATA 01H05BMCP13JF1FE0M3ZE6PXY1]" duration=2.36819672s shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01H05BMGG5C44354FEV6566GCK sources="[01H05BMF3DDHKRRT81YBE9PATA 01H05BMCP13JF1FE0M3ZE6PXY1]" duration=2.368408091s shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new="[01H05BMGG5XCYY9KDZKRNERBKF 01H05BMGG5C44354FEV6566GCK]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3102621333/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BMCP13JF1FE0M3ZE6PXY1 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3102621333/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BMF3DDHKRRT81YBE9PATA]" duration=2.368674506s duration_ms=2368 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3102621333/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BMGG5C44354FEV6566GCK/chunks/000001 dst=01H05BMGG5C44354FEV6566GCK/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3102621333/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3102621333/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BMGG5XCYY9KDZKRNERBKF/chunks/000001 dst=01H05BMGG5XCYY9KDZKRNERBKF/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3102621333/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3102621333/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BMGG5XCYY9KDZKRNERBKF/index dst=01H05BMGG5XCYY9KDZKRNERBKF/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3102621333/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01H05BMGG5XCYY9KDZKRNERBKF duration=3.787567ms duration_ms=3 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3102621333/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BMGG5C44354FEV6566GCK/index dst=01H05BMGG5C44354FEV6566GCK/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3102621333/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01H05BMGG5C44354FEV6566GCK duration=5.583346ms duration_ms=5 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=9.509642ms duration_ms=9 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BMCP13JF1FE0M3ZE6PXY1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BMCP13JF1FE0M3ZE6PXY1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BMF3DDHKRRT81YBE9PATA level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BMF3DDHKRRT81YBE9PATA level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=2.392551975s duration_ms=2392 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=5.034196ms duration_ms=5 cached=4 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=8.467664ms duration_ms=8 cached=4 returned=2 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=root1 ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=cleaner run_id=1683807165 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=9.573645ms level=info component=cleaner run_id=1683807165 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=17.995707ms duration_ms=18 cached=2 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job="stage: split, range start: 7200000, range end: 14400000, shard: 1_of_1, blocks: 01H05BMK3Q21A3PB44A1RGYS6A (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC),01H05BMMG5ASQZKWA65D2XT8NV (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01H05BMK3Q21A3PB44A1RGYS6A (min time: 7200000, max time: 14400000) 01H05BMMG5ASQZKWA65D2XT8NV (min time: 7200000, max time: 14400000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BMK3Q21A3PB44A1RGYS6A/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BMMG5ASQZKWA65D2XT8NV/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split2617392758/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BMK3Q21A3PB44A1RGYS6A /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split2617392758/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BMMG5ASQZKWA65D2XT8NV]" duration=14.461753ms duration_ms=14 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01H05BMQ8R4SYWXP9W02EVGQTA sources="[01H05BMK3Q21A3PB44A1RGYS6A 01H05BMMG5ASQZKWA65D2XT8NV]" duration=2.512238217s shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01H05BMQ8RPARS89V2EBKTKEPF sources="[01H05BMK3Q21A3PB44A1RGYS6A 01H05BMMG5ASQZKWA65D2XT8NV]" duration=2.51245967s shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new="[01H05BMQ8R4SYWXP9W02EVGQTA 01H05BMQ8RPARS89V2EBKTKEPF]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split2617392758/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BMK3Q21A3PB44A1RGYS6A /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split2617392758/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BMMG5ASQZKWA65D2XT8NV]" duration=2.512729444s duration_ms=2512 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split2617392758/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BMQ8R4SYWXP9W02EVGQTA/chunks/000001 dst=01H05BMQ8R4SYWXP9W02EVGQTA/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split2617392758/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split2617392758/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BMQ8RPARS89V2EBKTKEPF/chunks/000001 dst=01H05BMQ8RPARS89V2EBKTKEPF/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split2617392758/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split2617392758/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BMQ8R4SYWXP9W02EVGQTA/index dst=01H05BMQ8R4SYWXP9W02EVGQTA/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split2617392758/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01H05BMQ8R4SYWXP9W02EVGQTA duration=3.133132ms duration_ms=3 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split2617392758/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BMQ8RPARS89V2EBKTKEPF/index dst=01H05BMQ8RPARS89V2EBKTKEPF/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split2617392758/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01H05BMQ8RPARS89V2EBKTKEPF duration=4.256233ms duration_ms=4 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=7.162473ms duration_ms=7 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BMK3Q21A3PB44A1RGYS6A level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BMK3Q21A3PB44A1RGYS6A level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BMMG5ASQZKWA65D2XT8NV level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BMMG5ASQZKWA65D2XT8NV level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=2.540885388s duration_ms=2540 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=7.224076ms duration_ms=7 cached=4 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=8.671516ms duration_ms=8 cached=4 returned=2 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor --- FAIL: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor (151.03s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together (10.05s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/should_correctly_handle_empty_blocks_generated_in_the_splitting_stage (3.71s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/splitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0 (30.71s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split (11.01s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/non-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split (8.77s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/non-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split (7.35s) --- FAIL: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/smaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges (35.13s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together (11.65s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/splitting_should_be_disabled_if_configured_shards_=_0 (9.56s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/compaction_on_blocks_containing_native_histograms (9.42s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split (6.58s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split (7.08s) === RUN TestMultitenantCompactor_ShouldGuaranteeSeriesShardingConsistencyOverTheTime level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=root1 ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=cleaner run_id=1683807169 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=10.069672ms level=info component=cleaner run_id=1683807169 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=9.349472ms duration_ms=9 cached=1 returned=1 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job="stage: split, range start: 7200000, range end: 14400000, shard: 1_of_1, blocks: 01H05BMT0H6YZY8A79Z2F6TJDS (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=1 plan="[01H05BMT0H6YZY8A79Z2F6TJDS (min time: 7200000, max time: 14400000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01H05BMT0H6YZY8A79Z2F6TJDS/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=1 plan=[/tmp/TestMultitenantCompactor_ShouldGuaranteeSeriesShardingConsistencyOverTheTime2129023173/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BMT0H6YZY8A79Z2F6TJDS] duration=6.488356ms duration_ms=6 level=info component=compactor msg="compact blocks" count=1 mint=7200000 maxt=14400000 ulid=01H05BMVEGE0MEEBZ4X1MHPR3W sources=[01H05BMT0H6YZY8A79Z2F6TJDS] duration=4.971521851s shard=1_of_2 level=info component=compactor msg="compact blocks" count=1 mint=7200000 maxt=14400000 ulid=01H05BMVEGZSTKNYEVP7BFZ3MH sources=[01H05BMT0H6YZY8A79Z2F6TJDS] duration=4.971968916s shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new="[01H05BMVEGE0MEEBZ4X1MHPR3W 01H05BMVEGZSTKNYEVP7BFZ3MH]" blocks=[/tmp/TestMultitenantCompactor_ShouldGuaranteeSeriesShardingConsistencyOverTheTime2129023173/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BMT0H6YZY8A79Z2F6TJDS] duration=4.97240814s duration_ms=4972 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldGuaranteeSeriesShardingConsistencyOverTheTime2129023173/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BMVEGZSTKNYEVP7BFZ3MH/chunks/000001 dst=01H05BMVEGZSTKNYEVP7BFZ3MH/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldGuaranteeSeriesShardingConsistencyOverTheTime2129023173/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldGuaranteeSeriesShardingConsistencyOverTheTime2129023173/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BMVEGE0MEEBZ4X1MHPR3W/chunks/000001 dst=01H05BMVEGE0MEEBZ4X1MHPR3W/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldGuaranteeSeriesShardingConsistencyOverTheTime2129023173/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldGuaranteeSeriesShardingConsistencyOverTheTime2129023173/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BMVEGZSTKNYEVP7BFZ3MH/index dst=01H05BMVEGZSTKNYEVP7BFZ3MH/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldGuaranteeSeriesShardingConsistencyOverTheTime2129023173/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01H05BMVEGZSTKNYEVP7BFZ3MH duration=3.165293ms duration_ms=3 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldGuaranteeSeriesShardingConsistencyOverTheTime2129023173/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01H05BMVEGE0MEEBZ4X1MHPR3W/index dst=01H05BMVEGE0MEEBZ4X1MHPR3W/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldGuaranteeSeriesShardingConsistencyOverTheTime2129023173/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01H05BMVEGE0MEEBZ4X1MHPR3W duration=3.961657ms duration_ms=3 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=6.873817ms duration_ms=6 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01H05BMT0H6YZY8A79Z2F6TJDS level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01H05BMT0H6YZY8A79Z2F6TJDS level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=4.991687438s duration_ms=4991 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=4.184069ms duration_ms=4 cached=3 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=5.882403ms duration_ms=5 cached=3 returned=2 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor --- PASS: TestMultitenantCompactor_ShouldGuaranteeSeriesShardingConsistencyOverTheTime (6.72s) === RUN TestPlanCompaction === RUN TestPlanCompaction/should_split_a_single_block_if_==_smallest_compaction_range split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 20, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/should_split_a_single_block_if_<_smallest_compaction_range split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 20, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_in_different_time_ranges,_single_split_group split_merge_grouper_test.go:515: got 2 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 10, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_1, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/a_block_with_time_range_crossing_two_1st_level_ranges_should_be_NOT_considered_for_1st_level_splitting split_merge_grouper_test.go:515: got 2 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 20, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 20, range end: 40, shard: 1_of_1, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00.02 +0000 UTC, max time: 1970-01-01 00:00:00.03 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.03 +0000 UTC, max time: 1970-01-01 00:00:00.04 +0000 UTC) === RUN TestPlanCompaction/a_block_with_time_range_crossing_two_1st_level_ranges_should_BE_considered_for_2nd_level_compaction split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 0, range end: 40, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.03 +0000 UTC),00000000030000000000000000 (min time: 1970-01-01 00:00:00.02 +0000 UTC, max time: 1970-01-01 00:00:00.04 +0000 UTC) === RUN TestPlanCompaction/should_not_merge_blocks_within_the_same_time_range_but_with_different_external_labels split_merge_grouper_test.go:515: got 3 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_1, blocks: 00000000020000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC),00000000030000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_1, blocks: 00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/no_input_blocks split_merge_grouper_test.go:515: got 0 jobs: === RUN TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_within_the_same_time_range split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/should_merge_split_blocks_that_can_be_compacted_on_the_2nd_range_only split_merge_grouper_test.go:515: got 3 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 0, range end: 20, shard: 1_of_2, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: merge, range start: 0, range end: 20, shard: 2_of_2, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: merge, range start: 20, range end: 40, shard: 1_of_2, blocks: 00000000050000000000000000 (min time: 1970-01-01 00:00:00.02 +0000 UTC, max time: 1970-01-01 00:00:00.03 +0000 UTC),00000000060000000000000000 (min time: 1970-01-01 00:00:00.03 +0000 UTC, max time: 1970-01-01 00:00:00.04 +0000 UTC) === RUN TestPlanCompaction/should_not_split_non-split_blocks_if_they're_>_smallest_compaction_range_(do_not_split_historical_blocks_after_enabling_splitting) split_merge_grouper_test.go:515: got 2 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 0, range end: 20, shard: 1_of_2, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: merge, range start: 0, range end: 20, shard: 2_of_2, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/input_blocks_have_already_been_compacted_with_the_largest_range split_merge_grouper_test.go:515: got 0 jobs: === RUN TestPlanCompaction/a_range_containing_the_most_recent_block_shouldn't_be_prematurely_compacted_if_doesn't_cover_the_full_range split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 10, shard: 1_of_1, blocks: 00000000000000000000000000 (min time: 1970-01-01 00:00:00.005 +0000 UTC, max time: 1970-01-01 00:00:00.008 +0000 UTC),00000000000000000000000000 (min time: 1970-01-01 00:00:00.007 +0000 UTC, max time: 1970-01-01 00:00:00.009 +0000 UTC) === RUN TestPlanCompaction/should_NOT_split_a_single_block_if_==_smallest_compaction_range_but_configured_shards_=_0 split_merge_grouper_test.go:515: got 0 jobs: === RUN TestPlanCompaction/input_blocks_match_the_largest_range_but_can_be_compacted_because_overlapping split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 80, range end: 120, shard: 1_of_1, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00.08 +0000 UTC, max time: 1970-01-01 00:00:00.12 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.08 +0000 UTC, max time: 1970-01-01 00:00:00.12 +0000 UTC) === RUN TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_in_different_time_ranges,_two_split_groups split_merge_grouper_test.go:515: got 4 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 10, shard: 1_of_2, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 10, shard: 2_of_2, blocks: 00000000020000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_2, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 2_of_2, blocks: 00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/should_merge_but_NOT_split_multiple_1st_level_blocks_in_different_time_ranges_if_configured_shards_=_0 split_merge_grouper_test.go:515: got 2 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 0, range end: 10, shard: , blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC) split_merge_grouper_test.go:517: - stage: merge, range start: 10, range end: 20, shard: , blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/input_blocks_can_be_compacted_on_a_mix_of_1st_and_2nd_ranges,_guaranteeing_no_overlaps_and_giving_preference_to_smaller_ranges split_merge_grouper_test.go:515: got 3 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 20, range end: 40, shard: 1_of_1, blocks: 00000000040000000000000000 (min time: 1970-01-01 00:00:00.02 +0000 UTC, max time: 1970-01-01 00:00:00.03 +0000 UTC),00000000050000000000000000 (min time: 1970-01-01 00:00:00.03 +0000 UTC, max time: 1970-01-01 00:00:00.04 +0000 UTC) split_merge_grouper_test.go:517: - stage: merge, range start: 70, range end: 80, shard: 1_of_1, blocks: 00000000080000000000000000 (min time: 1970-01-01 00:00:00.07 +0000 UTC, max time: 1970-01-01 00:00:00.08 +0000 UTC),00000000090000000000000000 (min time: 1970-01-01 00:00:00.075 +0000 UTC, max time: 1970-01-01 00:00:00.08 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 10, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00.007 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC) === RUN TestPlanCompaction/a_block_with_time_range_larger_then_the_largest_compaction_range_should_NOT_be_considered_for_compaction split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 80, range end: 120, shard: 1_of_1, blocks: 00000000040000000000000000 (min time: 1970-01-01 00:00:00.08 +0000 UTC, max time: 1970-01-01 00:00:00.12 +0000 UTC),00000000050000000000000000 (min time: 1970-01-01 00:00:00.08 +0000 UTC, max time: 1970-01-01 00:00:00.12 +0000 UTC) === RUN TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_in_different_time_ranges split_merge_grouper_test.go:515: got 2 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 10, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_1, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) --- PASS: TestPlanCompaction (0.03s) --- PASS: TestPlanCompaction/should_split_a_single_block_if_==_smallest_compaction_range (0.00s) --- PASS: TestPlanCompaction/should_split_a_single_block_if_<_smallest_compaction_range (0.00s) --- PASS: TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_in_different_time_ranges,_single_split_group (0.00s) --- PASS: TestPlanCompaction/a_block_with_time_range_crossing_two_1st_level_ranges_should_be_NOT_considered_for_1st_level_splitting (0.00s) --- PASS: TestPlanCompaction/a_block_with_time_range_crossing_two_1st_level_ranges_should_BE_considered_for_2nd_level_compaction (0.00s) --- PASS: TestPlanCompaction/should_not_merge_blocks_within_the_same_time_range_but_with_different_external_labels (0.00s) --- PASS: TestPlanCompaction/no_input_blocks (0.00s) --- PASS: TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_within_the_same_time_range (0.00s) --- PASS: TestPlanCompaction/should_merge_split_blocks_that_can_be_compacted_on_the_2nd_range_only (0.00s) --- PASS: TestPlanCompaction/should_not_split_non-split_blocks_if_they're_>_smallest_compaction_range_(do_not_split_historical_blocks_after_enabling_splitting) (0.00s) --- PASS: TestPlanCompaction/input_blocks_have_already_been_compacted_with_the_largest_range (0.00s) --- PASS: TestPlanCompaction/a_range_containing_the_most_recent_block_shouldn't_be_prematurely_compacted_if_doesn't_cover_the_full_range (0.00s) --- PASS: TestPlanCompaction/should_NOT_split_a_single_block_if_==_smallest_compaction_range_but_configured_shards_=_0 (0.00s) --- PASS: TestPlanCompaction/input_blocks_match_the_largest_range_but_can_be_compacted_because_overlapping (0.00s) --- PASS: TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_in_different_time_ranges,_two_split_groups (0.00s) --- PASS: TestPlanCompaction/should_merge_but_NOT_split_multiple_1st_level_blocks_in_different_time_ranges_if_configured_shards_=_0 (0.00s) --- PASS: TestPlanCompaction/input_blocks_can_be_compacted_on_a_mix_of_1st_and_2nd_ranges,_guaranteeing_no_overlaps_and_giving_preference_to_smaller_ranges (0.00s) --- PASS: TestPlanCompaction/a_block_with_time_range_larger_then_the_largest_compaction_range_should_NOT_be_considered_for_compaction (0.00s) --- PASS: TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_in_different_time_ranges (0.00s) === RUN TestPlanSplitting === RUN TestPlanSplitting/should_return_nil_if_the_input_group_is_empty === RUN TestPlanSplitting/should_return_nil_if_the_input_group_contains_no_non-sharded_blocks === RUN TestPlanSplitting/should_return_a_split_job_if_the_input_group_contains_1_non-sharded_block === RUN TestPlanSplitting/should_splitGroups_split_jobs_if_the_input_group_contains_multiple_non-sharded_blocks --- PASS: TestPlanSplitting (0.00s) --- PASS: TestPlanSplitting/should_return_nil_if_the_input_group_is_empty (0.00s) --- PASS: TestPlanSplitting/should_return_nil_if_the_input_group_contains_no_non-sharded_blocks (0.00s) --- PASS: TestPlanSplitting/should_return_a_split_job_if_the_input_group_contains_1_non-sharded_block (0.00s) --- PASS: TestPlanSplitting/should_splitGroups_split_jobs_if_the_input_group_contains_multiple_non-sharded_blocks (0.00s) === RUN TestGroupBlocksByShardID === RUN TestGroupBlocksByShardID/no_input_blocks === RUN TestGroupBlocksByShardID/only_1_block_in_input_with_shard_ID === RUN TestGroupBlocksByShardID/only_1_block_in_input_without_shard_ID === RUN TestGroupBlocksByShardID/multiple_blocks_per_shard_ID --- PASS: TestGroupBlocksByShardID (0.00s) --- PASS: TestGroupBlocksByShardID/no_input_blocks (0.00s) --- PASS: TestGroupBlocksByShardID/only_1_block_in_input_with_shard_ID (0.00s) --- PASS: TestGroupBlocksByShardID/only_1_block_in_input_without_shard_ID (0.00s) --- PASS: TestGroupBlocksByShardID/multiple_blocks_per_shard_ID (0.00s) === RUN TestGroupBlocksByRange === RUN TestGroupBlocksByRange/no_input_blocks === RUN TestGroupBlocksByRange/only_1_block_in_input === RUN TestGroupBlocksByRange/only_1_block_per_range === RUN TestGroupBlocksByRange/multiple_blocks_per_range === RUN TestGroupBlocksByRange/a_block_with_time_range_larger_then_the_range_should_be_excluded === RUN TestGroupBlocksByRange/blocks_with_different_time_ranges_but_all_fitting_within_the_input_range --- PASS: TestGroupBlocksByRange (0.00s) --- PASS: TestGroupBlocksByRange/no_input_blocks (0.00s) --- PASS: TestGroupBlocksByRange/only_1_block_in_input (0.00s) --- PASS: TestGroupBlocksByRange/only_1_block_per_range (0.00s) --- PASS: TestGroupBlocksByRange/multiple_blocks_per_range (0.00s) --- PASS: TestGroupBlocksByRange/a_block_with_time_range_larger_then_the_range_should_be_excluded (0.00s) --- PASS: TestGroupBlocksByRange/blocks_with_different_time_ranges_but_all_fitting_within_the_input_range (0.00s) === RUN TestJob_conflicts === RUN TestJob_conflicts/should_conflict_if_jobs_compact_different_blocks_but_with_overlapping_time_ranges_and_same_shard === RUN TestJob_conflicts/should_NOT_conflict_if_jobs_compact_different_blocks_with_non-overlapping_time_ranges_and_same_shard === RUN TestJob_conflicts/should_NOT_conflict_if_jobs_compact_same_blocks_with_overlapping_time_ranges_but_different_shard === RUN TestJob_conflicts/should_conflict_if_jobs_compact_same_blocks_with_overlapping_time_ranges_and_different_shard_but_at_a_different_stage === RUN TestJob_conflicts/should_conflict_between_split_and_merge_jobs_with_overlapping_time_ranges === RUN TestJob_conflicts/should_NOT_conflict_between_split_and_merge_jobs_with_non-overlapping_time_ranges --- PASS: TestJob_conflicts (0.01s) --- PASS: TestJob_conflicts/should_conflict_if_jobs_compact_different_blocks_but_with_overlapping_time_ranges_and_same_shard (0.00s) --- PASS: TestJob_conflicts/should_NOT_conflict_if_jobs_compact_different_blocks_with_non-overlapping_time_ranges_and_same_shard (0.00s) --- PASS: TestJob_conflicts/should_NOT_conflict_if_jobs_compact_same_blocks_with_overlapping_time_ranges_but_different_shard (0.00s) --- PASS: TestJob_conflicts/should_conflict_if_jobs_compact_same_blocks_with_overlapping_time_ranges_and_different_shard_but_at_a_different_stage (0.00s) --- PASS: TestJob_conflicts/should_conflict_between_split_and_merge_jobs_with_overlapping_time_ranges (0.00s) --- PASS: TestJob_conflicts/should_NOT_conflict_between_split_and_merge_jobs_with_non-overlapping_time_ranges (0.00s) === RUN TestBlocksGroup_overlaps --- PASS: TestBlocksGroup_overlaps (0.00s) === RUN TestBlocksGroup_getNonShardedBlocks --- PASS: TestBlocksGroup_getNonShardedBlocks (0.00s) === RUN TestSplitAndMergePlanner_Plan === RUN TestSplitAndMergePlanner_Plan/no_blocks === RUN TestSplitAndMergePlanner_Plan/a_source_block_is_larger_then_the_largest_range === RUN TestSplitAndMergePlanner_Plan/source_blocks_are_smaller_then_the_largest_range_but_compacted_block_is_larger === RUN TestSplitAndMergePlanner_Plan/source_blocks_and_compacted_block_are_smaller_then_the_largest_range_but_misaligned === RUN TestSplitAndMergePlanner_Plan/blocks_fit_within_the_largest_range --- PASS: TestSplitAndMergePlanner_Plan (0.00s) --- PASS: TestSplitAndMergePlanner_Plan/no_blocks (0.00s) --- PASS: TestSplitAndMergePlanner_Plan/a_source_block_is_larger_then_the_largest_range (0.00s) --- PASS: TestSplitAndMergePlanner_Plan/source_blocks_are_smaller_then_the_largest_range_but_compacted_block_is_larger (0.00s) --- PASS: TestSplitAndMergePlanner_Plan/source_blocks_and_compacted_block_are_smaller_then_the_largest_range_but_misaligned (0.00s) --- PASS: TestSplitAndMergePlanner_Plan/blocks_fit_within_the_largest_range (0.00s) === RUN TestSyncerMetrics --- PASS: TestSyncerMetrics (0.04s) === RUN TestDeleteTenant --- PASS: TestDeleteTenant (0.16s) === RUN TestDeleteTenantStatus === RUN TestDeleteTenantStatus/empty === RUN TestDeleteTenantStatus/no_user_objects === RUN TestDeleteTenantStatus/non-block_files === RUN TestDeleteTenantStatus/block_files --- PASS: TestDeleteTenantStatus (0.63s) --- PASS: TestDeleteTenantStatus/empty (0.15s) --- PASS: TestDeleteTenantStatus/no_user_objects (0.15s) --- PASS: TestDeleteTenantStatus/non-block_files (0.16s) --- PASS: TestDeleteTenantStatus/block_files (0.16s) === CONT TestSyncer_GarbageCollect_e2e === RUN TestSyncer_GarbageCollect_e2e/inmem create 00000000000000000000000000 create 00000000010000000000000000 create 00000000020000000000000000 create 00000000030000000000000000 create 00000000040000000000000000 create 00000000050000000000000000 create 00000000060000000000000000 create 00000000070000000000000000 create 00000000080000000000000000 create 00000000090000000000000000 create 00000000340000000000000000 create 00000000680000000000000000 === CONT TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForDeletion create 000000009C0000000000000000 create 00000000CG0000000000000000 create 00000000FM0000000000000000 === RUN TestSyncer_GarbageCollect_e2e/filesystem === PAUSE TestSyncer_GarbageCollect_e2e/filesystem === CONT TestMultitenantCompactor_ShouldFailCompactionOnTimeout --- PASS: TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForDeletion (0.07s) === CONT TestMultitenantCompactor_ShouldSkipCompactionForJobsWithFirstLevelCompactionBlocksAndWaitPeriodNotElapsed --- PASS: TestMultitenantCompactor_ShouldFailCompactionOnTimeout (5.01s) === CONT TestMultitenantCompactor_ShouldSkipCompactionForJobsNoMoreOwnedAfterPlanning --- PASS: TestMultitenantCompactor_ShouldSkipCompactionForJobsWithFirstLevelCompactionBlocksAndWaitPeriodNotElapsed (5.16s) === CONT TestMultitenantCompactor_ShouldCompactOnlyUsersOwnedByTheInstanceOnShardingEnabledAndMultipleInstancesRunning --- PASS: TestMultitenantCompactor_ShouldSkipCompactionForJobsNoMoreOwnedAfterPlanning (0.25s) === CONT TestMultitenantCompactor_ShouldCompactAllUsersOnShardingEnabledButOnlyOneInstanceRunning --- PASS: TestMultitenantCompactor_ShouldCompactAllUsersOnShardingEnabledButOnlyOneInstanceRunning (0.16s) === CONT TestMultitenantCompactor_ShouldNotCompactBlocksForUsersMarkedForDeletion --- PASS: TestMultitenantCompactor_ShouldNotCompactBlocksForUsersMarkedForDeletion (0.08s) === CONT TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForNoCompaction --- PASS: TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForNoCompaction (0.23s) === CONT TestMultitenantCompactor_ShouldIncrementCompactionErrorIfFailedToCompactASingleTenant --- PASS: TestMultitenantCompactor_ShouldIncrementCompactionErrorIfFailedToCompactASingleTenant (0.82s) === CONT TestMultitenantCompactor_ShouldStopCompactingTenantOnReachingMaxCompactionTime --- PASS: TestMultitenantCompactor_ShouldStopCompactingTenantOnReachingMaxCompactionTime (1.25s) === CONT TestMultitenantCompactor_ShouldIterateOverUsersAndRunCompaction --- PASS: TestMultitenantCompactor_ShouldIterateOverUsersAndRunCompaction (0.29s) === CONT TestMultitenantCompactor_ShouldIncrementCompactionShutdownIfTheContextIsCancelled compactor_test.go:507: level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=root1 ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=test-user level=info component=compactor user=test-user msg="start sync of metas" level=debug component=compactor user=test-user component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=test-user component=block.BaseFetcher msg="successfully synchronized block metadata" cached=2 returned=2 partial=0 level=info component=compactor user=test-user msg="start of GC" level=debug component=compactor user=test-user msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge--1574776800000-1574784000000 job="stage: merge, range start: 1574776800000, range end: 1574784000000, shard: , blocks: 01DTVP434PA9VFXSW2JKB3392D (min time: 2019-11-26 14:00:00 +0000 UTC, max time: 2019-11-26 16:00:00 +0000 UTC),01DTW0ZCPDDNV4BV83Q2SV4QAZ (min time: 2019-11-26 14:00:00 +0000 UTC, max time: 2019-11-26 16:00:00 +0000 UTC)" level=info component=compactor user=test-user msg="start of compactions" level=error component=compactor user=test-user groupKey=0@17241709254077376921-merge--1574776800000-1574784000000 msg="compaction job failed" err="plan compaction: context canceled" level=info component=compactor user=test-user msg="start sync of metas" level=debug component=compactor user=test-user component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=test-user component=block.BaseFetcher msg="successfully synchronized block metadata" cached=2 returned=2 partial=0 level=info component=compactor user=test-user msg="start of GC" level=debug component=compactor user=test-user msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge--1574776800000-1574784000000 job="stage: merge, range start: 1574776800000, range end: 1574784000000, shard: , blocks: 01DTVP434PA9VFXSW2JKB3392D (min time: 2019-11-26 14:00:00 +0000 UTC, max time: 2019-11-26 16:00:00 +0000 UTC),01DTW0ZCPDDNV4BV83Q2SV4QAZ (min time: 2019-11-26 14:00:00 +0000 UTC, max time: 2019-11-26 16:00:00 +0000 UTC)" level=info component=compactor user=test-user msg="start of compactions" level=error component=compactor user=test-user groupKey=0@17241709254077376921-merge--1574776800000-1574784000000 msg="compaction job failed" err="plan compaction: context canceled" level=info component=compactor user=test-user msg="start sync of metas" level=debug component=compactor user=test-user component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=test-user component=block.BaseFetcher msg="successfully synchronized block metadata" cached=2 returned=2 partial=0 level=info component=compactor user=test-user msg="start of GC" level=debug component=compactor user=test-user msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge--1574776800000-1574784000000 job="stage: merge, range start: 1574776800000, range end: 1574784000000, shard: , blocks: 01DTVP434PA9VFXSW2JKB3392D (min time: 2019-11-26 14:00:00 +0000 UTC, max time: 2019-11-26 16:00:00 +0000 UTC),01DTW0ZCPDDNV4BV83Q2SV4QAZ (min time: 2019-11-26 14:00:00 +0000 UTC, max time: 2019-11-26 16:00:00 +0000 UTC)" level=info component=compactor user=test-user msg="start of compactions" level=error component=compactor user=test-user groupKey=0@17241709254077376921-merge--1574776800000-1574784000000 msg="compaction job failed" err="plan compaction: context canceled" level=info component=compactor msg="compaction for user was interrupted by a shutdown" user=test-user level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor --- PASS: TestMultitenantCompactor_ShouldIncrementCompactionShutdownIfTheContextIsCancelled (0.21s) === CONT TestMultitenantCompactor_ShouldDoNothingOnNoUserBlocks --- PASS: TestMultitenantCompactor_ShouldDoNothingOnNoUserBlocks (0.21s) === CONT TestMultitenantCompactor_ShouldRetryCompactionOnFailureWhileDiscoveringUsersFromBucket compactor_test.go:307: level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=root1 ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=error component=compactor msg="failed to discover users from bucket" err="failed to iterate the bucket" level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor --- PASS: TestMultitenantCompactor_ShouldRetryCompactionOnFailureWhileDiscoveringUsersFromBucket (0.04s) === CONT TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly === RUN TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly/inmem create 00000000000000000000000000 create 00000000010000000000000000 create 00000000340000000000000000 level=debug msg="deleted file" file=00000000000000000000000000/meta.json bucket=inmem level=debug msg="deleted file" file=00000000000000000000000000/deletion-mark.json bucket=inmem level=debug msg="deleted file" file=00000000010000000000000000/meta.json bucket=inmem level=debug msg="deleted file" file=00000000010000000000000000/deletion-mark.json bucket=inmem === RUN TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly/filesystem === PAUSE TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly/filesystem === CONT TestGroupCompactE2E === RUN TestGroupCompactE2E/inmem level=info msg="start sync of metas" level=info msg="start of GC" level=info msg="start of compactions" level=info msg="compaction iterations done" bucket_compactor_e2e_test.go:476: Error Trace: /build/mimir/src/mimir/pkg/compactor/bucket_compactor_e2e_test.go:476 /build/mimir/src/mimir/pkg/compactor/bucket_compactor_e2e_test.go:452 /build/mimir/src/mimir/pkg/compactor/bucket_compactor_e2e_test.go:270 /build/mimir/src/mimir/pkg/compactor/bucket_compactor_e2e_test.go:592 Error: Received unexpected error: write block: populate block: writing block: closing index writer: context deadline exceeded Test: TestGroupCompactE2E/inmem --- FAIL: TestGroupCompactE2E (20.38s) --- FAIL: TestGroupCompactE2E/inmem (20.38s) === CONT TestSyncer_GarbageCollect_e2e/filesystem create 00000000000000000000000000 create 00000000010000000000000000 create 00000000020000000000000000 create 00000000030000000000000000 create 00000000040000000000000000 create 00000000050000000000000000 create 00000000060000000000000000 create 00000000070000000000000000 create 00000000080000000000000000 create 00000000090000000000000000 create 00000000340000000000000000 create 00000000680000000000000000 create 000000009C0000000000000000 create 00000000CG0000000000000000 create 00000000FM0000000000000000 --- PASS: TestSyncer_GarbageCollect_e2e (0.03s) --- PASS: TestSyncer_GarbageCollect_e2e/inmem (0.03s) --- PASS: TestSyncer_GarbageCollect_e2e/filesystem (0.04s) === CONT TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly/filesystem create 00000000000000000000000000 create 00000000010000000000000000 create 00000000340000000000000000 level=debug msg="deleted file" file=00000000000000000000000000/meta.json bucket="fs: /tmp/TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnlyfilesystem2870394063/001" level=debug msg="deleted file" file=00000000000000000000000000/deletion-mark.json bucket="fs: /tmp/TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnlyfilesystem2870394063/001" level=debug msg="deleted file" file=00000000010000000000000000/meta.json bucket="fs: /tmp/TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnlyfilesystem2870394063/001" level=debug msg="deleted file" file=00000000010000000000000000/deletion-mark.json bucket="fs: /tmp/TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnlyfilesystem2870394063/001" --- PASS: TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly (0.01s) --- PASS: TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly/inmem (0.01s) --- PASS: TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly/filesystem (0.02s) === NAME TestMultitenantCompactor_ShouldCompactOnlyUsersOwnedByTheInstanceOnShardingEnabledAndMultipleInstancesRunning compactor_test.go:1222: expected true, got false --- FAIL: TestMultitenantCompactor_ShouldCompactOnlyUsersOwnedByTheInstanceOnShardingEnabledAndMultipleInstancesRunning (36.79s) FAIL FAIL github.com/grafana/mimir/pkg/compactor 319.094s === RUN TestClient_WriteSeries === RUN TestClient_WriteSeries/write_series_in_a_single_batch === RUN TestClient_WriteSeries/write_series_in_multiple_batches === RUN TestClient_WriteSeries/request_failed_with_4xx_error === RUN TestClient_WriteSeries/request_failed_with_5xx_error --- PASS: TestClient_WriteSeries (0.02s) --- PASS: TestClient_WriteSeries/write_series_in_a_single_batch (0.01s) --- PASS: TestClient_WriteSeries/write_series_in_multiple_batches (0.00s) --- PASS: TestClient_WriteSeries/request_failed_with_4xx_error (0.00s) --- PASS: TestClient_WriteSeries/request_failed_with_5xx_error (0.00s) === RUN TestClient_QueryRange === RUN TestClient_QueryRange/results_cache_not_explicitly_disabled === RUN TestClient_QueryRange/results_cache_disabled --- PASS: TestClient_QueryRange (0.01s) --- PASS: TestClient_QueryRange/results_cache_not_explicitly_disabled (0.01s) --- PASS: TestClient_QueryRange/results_cache_disabled (0.00s) === RUN TestClient_Query === RUN TestClient_Query/results_cache_not_explicitly_disabled === RUN TestClient_Query/results_cache_disabled --- PASS: TestClient_Query (0.01s) --- PASS: TestClient_Query/results_cache_not_explicitly_disabled (0.00s) --- PASS: TestClient_Query/results_cache_disabled (0.00s) === RUN TestManager_PeriodicRun --- PASS: TestManager_PeriodicRun (0.05s) === RUN TestManager_SmokeTest === RUN TestManager_SmokeTest/successful_smoke_test === RUN TestManager_SmokeTest/failed_smoke_test --- PASS: TestManager_SmokeTest (0.00s) --- PASS: TestManager_SmokeTest/successful_smoke_test (0.00s) --- PASS: TestManager_SmokeTest/failed_smoke_test (0.00s) === RUN TestAlignTimestampToInterval --- PASS: TestAlignTimestampToInterval (0.00s) === RUN TestGetQueryStep === RUN TestGetQueryStep/should_return_write_interval_if_expected_number_of_samples_is_<_1000 === RUN TestGetQueryStep/should_align_step_to_write_interval_and_guarantee_no_more_than_1000_samples --- PASS: TestGetQueryStep (0.00s) --- PASS: TestGetQueryStep/should_return_write_interval_if_expected_number_of_samples_is_<_1000 (0.00s) --- PASS: TestGetQueryStep/should_align_step_to_write_interval_and_guarantee_no_more_than_1000_samples (0.00s) === RUN TestVerifySamplesSum === RUN TestVerifySamplesSum/generateSineWaveValue:should_return_no_error_if_all_samples_value_and_timestamp_match_the_expected_one_(1_series) === RUN TestVerifySamplesSum/generateSineWaveValue:should_return_no_error_if_all_samples_value_and_timestamp_match_the_expected_one_(multiple_series) === RUN TestVerifySamplesSum/generateSineWaveValue:should_return_error_if_there's_a_missing_series === RUN TestVerifySamplesSum/generateSineWaveValue:should_return_error_if_there's_a_missing_sample === RUN TestVerifySamplesSum/generateSineWaveValue:should_return_error_if_the_2nd_last_sample_has_an_unexpected_timestamp === RUN TestVerifySamplesSum/generateHistogramValue_for_histogram_int_counter:should_return_no_error_if_all_histograms_value_and_timestamp_match_the_expected_one_(1_series) === RUN TestVerifySamplesSum/generateHistogramValue_for_histogram_int_counter:should_return_no_error_if_all_histograms_value_and_timestamp_match_the_expected_one_(multiple_series) === RUN TestVerifySamplesSum/generateHistogramValue_for_histogram_int_counter:should_return_error_if_there's_a_missing_series === RUN TestVerifySamplesSum/generateHistogramValue_for_histogram_int_counter:should_return_error_if_there's_a_missing_histogram === RUN TestVerifySamplesSum/generateHistogramValue_for_histogram_int_counter:should_return_error_if_the_2nd_last_histogram_has_an_unexpected_timestamp === RUN TestVerifySamplesSum/generateHistogramValue_for_histogram_float_counter:should_return_no_error_if_all_histograms_value_and_timestamp_match_the_expected_one_(multiple_series) === RUN TestVerifySamplesSum/generateHistogramValue_for_histogram_float_counter:should_return_error_if_there's_a_missing_series === RUN TestVerifySamplesSum/generateHistogramValue_for_histogram_float_counter:should_return_error_if_there's_a_missing_histogram === RUN TestVerifySamplesSum/generateHistogramValue_for_histogram_float_counter:should_return_error_if_the_2nd_last_histogram_has_an_unexpected_timestamp === RUN TestVerifySamplesSum/generateHistogramValue_for_histogram_float_counter:should_return_no_error_if_all_histograms_value_and_timestamp_match_the_expected_one_(1_series) === RUN TestVerifySamplesSum/generateHistogramValue_for_histogram_int_gauge:should_return_error_if_the_2nd_last_histogram_has_an_unexpected_timestamp === RUN TestVerifySamplesSum/generateHistogramValue_for_histogram_int_gauge:should_return_no_error_if_all_histograms_value_and_timestamp_match_the_expected_one_(1_series) === RUN TestVerifySamplesSum/generateHistogramValue_for_histogram_int_gauge:should_return_no_error_if_all_histograms_value_and_timestamp_match_the_expected_one_(multiple_series) === RUN TestVerifySamplesSum/generateHistogramValue_for_histogram_int_gauge:should_return_error_if_there's_a_missing_series === RUN TestVerifySamplesSum/generateHistogramValue_for_histogram_int_gauge:should_return_error_if_there's_a_missing_histogram === RUN TestVerifySamplesSum/generateHistogramValue_for_histogram_float_gauge:should_return_error_if_the_2nd_last_histogram_has_an_unexpected_timestamp === RUN TestVerifySamplesSum/generateHistogramValue_for_histogram_float_gauge:should_return_no_error_if_all_histograms_value_and_timestamp_match_the_expected_one_(1_series) === RUN TestVerifySamplesSum/generateHistogramValue_for_histogram_float_gauge:should_return_no_error_if_all_histograms_value_and_timestamp_match_the_expected_one_(multiple_series) === RUN TestVerifySamplesSum/generateHistogramValue_for_histogram_float_gauge:should_return_error_if_there's_a_missing_series === RUN TestVerifySamplesSum/generateHistogramValue_for_histogram_float_gauge:should_return_error_if_there's_a_missing_histogram --- PASS: TestVerifySamplesSum (0.05s) --- PASS: TestVerifySamplesSum/generateSineWaveValue:should_return_no_error_if_all_samples_value_and_timestamp_match_the_expected_one_(1_series) (0.00s) --- PASS: TestVerifySamplesSum/generateSineWaveValue:should_return_no_error_if_all_samples_value_and_timestamp_match_the_expected_one_(multiple_series) (0.00s) --- PASS: TestVerifySamplesSum/generateSineWaveValue:should_return_error_if_there's_a_missing_series (0.00s) --- PASS: TestVerifySamplesSum/generateSineWaveValue:should_return_error_if_there's_a_missing_sample (0.00s) --- PASS: TestVerifySamplesSum/generateSineWaveValue:should_return_error_if_the_2nd_last_sample_has_an_unexpected_timestamp (0.00s) --- PASS: TestVerifySamplesSum/generateHistogramValue_for_histogram_int_counter:should_return_no_error_if_all_histograms_value_and_timestamp_match_the_expected_one_(1_series) (0.00s) --- PASS: TestVerifySamplesSum/generateHistogramValue_for_histogram_int_counter:should_return_no_error_if_all_histograms_value_and_timestamp_match_the_expected_one_(multiple_series) (0.00s) --- PASS: TestVerifySamplesSum/generateHistogramValue_for_histogram_int_counter:should_return_error_if_there's_a_missing_series (0.00s) --- PASS: TestVerifySamplesSum/generateHistogramValue_for_histogram_int_counter:should_return_error_if_there's_a_missing_histogram (0.00s) --- PASS: TestVerifySamplesSum/generateHistogramValue_for_histogram_int_counter:should_return_error_if_the_2nd_last_histogram_has_an_unexpected_timestamp (0.00s) --- PASS: TestVerifySamplesSum/generateHistogramValue_for_histogram_float_counter:should_return_no_error_if_all_histograms_value_and_timestamp_match_the_expected_one_(multiple_series) (0.00s) --- PASS: TestVerifySamplesSum/generateHistogramValue_for_histogram_float_counter:should_return_error_if_there's_a_missing_series (0.00s) --- PASS: TestVerifySamplesSum/generateHistogramValue_for_histogram_float_counter:should_return_error_if_there's_a_missing_histogram (0.00s) --- PASS: TestVerifySamplesSum/generateHistogramValue_for_histogram_float_counter:should_return_error_if_the_2nd_last_histogram_has_an_unexpected_timestamp (0.00s) --- PASS: TestVerifySamplesSum/generateHistogramValue_for_histogram_float_counter:should_return_no_error_if_all_histograms_value_and_timestamp_match_the_expected_one_(1_series) (0.00s) --- PASS: TestVerifySamplesSum/generateHistogramValue_for_histogram_int_gauge:should_return_error_if_the_2nd_last_histogram_has_an_unexpected_timestamp (0.00s) --- PASS: TestVerifySamplesSum/generateHistogramValue_for_histogram_int_gauge:should_return_no_error_if_all_histograms_value_and_timestamp_match_the_expected_one_(1_series) (0.00s) --- PASS: TestVerifySamplesSum/generateHistogramValue_for_histogram_int_gauge:should_return_no_error_if_all_histograms_value_and_timestamp_match_the_expected_one_(multiple_series) (0.00s) --- PASS: TestVerifySamplesSum/generateHistogramValue_for_histogram_int_gauge:should_return_error_if_there's_a_missing_series (0.00s) --- PASS: TestVerifySamplesSum/generateHistogramValue_for_histogram_int_gauge:should_return_error_if_there's_a_missing_histogram (0.00s) --- PASS: TestVerifySamplesSum/generateHistogramValue_for_histogram_float_gauge:should_return_error_if_the_2nd_last_histogram_has_an_unexpected_timestamp (0.00s) --- PASS: TestVerifySamplesSum/generateHistogramValue_for_histogram_float_gauge:should_return_no_error_if_all_histograms_value_and_timestamp_match_the_expected_one_(1_series) (0.00s) --- PASS: TestVerifySamplesSum/generateHistogramValue_for_histogram_float_gauge:should_return_no_error_if_all_histograms_value_and_timestamp_match_the_expected_one_(multiple_series) (0.00s) --- PASS: TestVerifySamplesSum/generateHistogramValue_for_histogram_float_gauge:should_return_error_if_there's_a_missing_series (0.00s) --- PASS: TestVerifySamplesSum/generateHistogramValue_for_histogram_float_gauge:should_return_error_if_there's_a_missing_histogram (0.00s) === RUN TestMinTime --- PASS: TestMinTime (0.00s) === RUN TestMaxTime --- PASS: TestMaxTime (0.00s) === RUN TestRandTime --- PASS: TestRandTime (0.01s) === RUN TestCompareSampleValues === RUN TestCompareSampleValues/histogram_float_counter_shouldn't_work_with_float_tolerance === RUN TestCompareSampleValues/histogram_float_counter_should_work_with_histogram_tolerance --- PASS: TestCompareSampleValues (0.00s) --- PASS: TestCompareSampleValues/histogram_float_counter_shouldn't_work_with_float_tolerance (0.00s) --- PASS: TestCompareSampleValues/histogram_float_counter_should_work_with_histogram_tolerance (0.00s) === RUN TestWriteReadSeriesTest_Run === RUN TestWriteReadSeriesTest_Run/should_write_series_with_current_timestamp_if_it's_already_aligned_to_write_interval === RUN TestWriteReadSeriesTest_Run/should_write_series_with_timestamp_aligned_to_write_interval === RUN TestWriteReadSeriesTest_Run/should_write_series_from_last_written_timestamp_until_now === RUN TestWriteReadSeriesTest_Run/should_stop_remote_writing_on_network_error === RUN TestWriteReadSeriesTest_Run/should_stop_remote_writing_on_5xx_error === RUN TestWriteReadSeriesTest_Run/should_keep_remote_writing_next_intervals_on_4xx_error === RUN TestWriteReadSeriesTest_Run/should_query_written_series,_compare_results_and_track_no_failure_if_results_match === RUN TestWriteReadSeriesTest_Run/should_query_written_series,_compare_results_and_track_failure_if_results_don't_match === RUN TestWriteReadSeriesTest_Run/should_write_series_with_current_timestamp_if_it's_already_aligned_to_write_interval#01 === RUN TestWriteReadSeriesTest_Run/should_write_series_with_timestamp_aligned_to_write_interval#01 === RUN TestWriteReadSeriesTest_Run/should_write_series_from_last_written_timestamp_until_now#01 === RUN TestWriteReadSeriesTest_Run/should_stop_remote_writing_on_network_error#01 === RUN TestWriteReadSeriesTest_Run/should_stop_remote_writing_on_5xx_error#01 === RUN TestWriteReadSeriesTest_Run/should_keep_remote_writing_next_intervals_on_4xx_error#01 === RUN TestWriteReadSeriesTest_Run/should_query_written_series,_compare_results_and_track_no_failure_if_results_match#01 === RUN TestWriteReadSeriesTest_Run/should_query_written_series,_compare_results_and_track_failure_if_results_don't_match#01 --- PASS: TestWriteReadSeriesTest_Run (44.89s) --- PASS: TestWriteReadSeriesTest_Run/should_write_series_with_current_timestamp_if_it's_already_aligned_to_write_interval (0.02s) --- PASS: TestWriteReadSeriesTest_Run/should_write_series_with_timestamp_aligned_to_write_interval (0.02s) --- PASS: TestWriteReadSeriesTest_Run/should_write_series_from_last_written_timestamp_until_now (2.02s) --- PASS: TestWriteReadSeriesTest_Run/should_stop_remote_writing_on_network_error (0.01s) --- PASS: TestWriteReadSeriesTest_Run/should_stop_remote_writing_on_5xx_error (0.00s) --- PASS: TestWriteReadSeriesTest_Run/should_keep_remote_writing_next_intervals_on_4xx_error (2.01s) --- PASS: TestWriteReadSeriesTest_Run/should_query_written_series,_compare_results_and_track_no_failure_if_results_match (0.01s) --- PASS: TestWriteReadSeriesTest_Run/should_query_written_series,_compare_results_and_track_failure_if_results_don't_match (0.01s) --- PASS: TestWriteReadSeriesTest_Run/should_write_series_with_current_timestamp_if_it's_already_aligned_to_write_interval#01 (3.06s) --- PASS: TestWriteReadSeriesTest_Run/should_write_series_with_timestamp_aligned_to_write_interval#01 (3.12s) --- PASS: TestWriteReadSeriesTest_Run/should_write_series_from_last_written_timestamp_until_now#01 (11.15s) --- PASS: TestWriteReadSeriesTest_Run/should_stop_remote_writing_on_network_error#01 (3.03s) --- PASS: TestWriteReadSeriesTest_Run/should_stop_remote_writing_on_5xx_error#01 (3.03s) --- PASS: TestWriteReadSeriesTest_Run/should_keep_remote_writing_next_intervals_on_4xx_error#01 (11.19s) --- PASS: TestWriteReadSeriesTest_Run/should_query_written_series,_compare_results_and_track_no_failure_if_results_match#01 (3.08s) --- PASS: TestWriteReadSeriesTest_Run/should_query_written_series,_compare_results_and_track_failure_if_results_don't_match#01 (3.13s) === RUN TestWriteReadSeriesTest_Init === RUN TestWriteReadSeriesTest_Init/no_previously_written_samples_found === RUN TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-2h,_-1m] === RUN TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-36h,_-1m] === RUN TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-36h,_-1m]_but_last_data_point_of_previous_24h_period_is_missing === RUN TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-24h,_-1m] === RUN TestWriteReadSeriesTest_Init/the_configured_query_max_age_is_>_24h === RUN TestWriteReadSeriesTest_Init/the_configured_query_max_age_is_<_24h === RUN TestWriteReadSeriesTest_Init/the_most_recent_previously_written_data_point_is_older_than_1h_ago === RUN TestWriteReadSeriesTest_Init/the_first_query_fails === RUN TestWriteReadSeriesTest_Init/a_subsequent_query_fails === RUN TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_in_the_middle_of_the_last_24h_period === RUN TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_in_the_middle_of_the_previous_24h_period === RUN TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_exactly_at_the_beginning_of_this_24h_period === RUN TestWriteReadSeriesTest_Init/no_previously_written_samples_found#01 === RUN TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-2h,_-1m]#01 === RUN TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-36h,_-1m]#01 === RUN TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-36h,_-1m]_but_last_data_point_of_previous_24h_period_is_missing#01 === RUN TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-24h,_-1m]#01 === RUN TestWriteReadSeriesTest_Init/the_configured_query_max_age_is_>_24h#01 === RUN TestWriteReadSeriesTest_Init/the_configured_query_max_age_is_<_24h#01 === RUN TestWriteReadSeriesTest_Init/the_most_recent_previously_written_data_point_is_older_than_1h_ago#01 === RUN TestWriteReadSeriesTest_Init/the_first_query_fails#01 === RUN TestWriteReadSeriesTest_Init/a_subsequent_query_fails#01 === RUN TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_in_the_middle_of_the_last_24h_period#01 === RUN TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_in_the_middle_of_the_previous_24h_period#01 === RUN TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_exactly_at_the_beginning_of_this_24h_period#01 --- PASS: TestWriteReadSeriesTest_Init (23.61s) --- PASS: TestWriteReadSeriesTest_Init/no_previously_written_samples_found (0.00s) --- PASS: TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-2h,_-1m] (0.00s) --- PASS: TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-36h,_-1m] (0.05s) --- PASS: TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-36h,_-1m]_but_last_data_point_of_previous_24h_period_is_missing (0.10s) --- PASS: TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-24h,_-1m] (0.04s) --- PASS: TestWriteReadSeriesTest_Init/the_configured_query_max_age_is_>_24h (0.05s) --- PASS: TestWriteReadSeriesTest_Init/the_configured_query_max_age_is_<_24h (0.00s) --- PASS: TestWriteReadSeriesTest_Init/the_most_recent_previously_written_data_point_is_older_than_1h_ago (0.00s) --- PASS: TestWriteReadSeriesTest_Init/the_first_query_fails (0.00s) --- PASS: TestWriteReadSeriesTest_Init/a_subsequent_query_fails (0.02s) --- PASS: TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_in_the_middle_of_the_last_24h_period (0.01s) --- PASS: TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_in_the_middle_of_the_previous_24h_period (0.08s) --- PASS: TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_exactly_at_the_beginning_of_this_24h_period (0.14s) --- PASS: TestWriteReadSeriesTest_Init/no_previously_written_samples_found#01 (0.01s) --- PASS: TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-2h,_-1m]#01 (0.20s) --- PASS: TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-36h,_-1m]#01 (2.89s) --- PASS: TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-36h,_-1m]_but_last_data_point_of_previous_24h_period_is_missing#01 (2.37s) --- PASS: TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-24h,_-1m]#01 (2.60s) --- PASS: TestWriteReadSeriesTest_Init/the_configured_query_max_age_is_>_24h#01 (6.67s) --- PASS: TestWriteReadSeriesTest_Init/the_configured_query_max_age_is_<_24h#01 (0.10s) --- PASS: TestWriteReadSeriesTest_Init/the_most_recent_previously_written_data_point_is_older_than_1h_ago#01 (0.06s) --- PASS: TestWriteReadSeriesTest_Init/the_first_query_fails#01 (0.01s) --- PASS: TestWriteReadSeriesTest_Init/a_subsequent_query_fails#01 (1.24s) --- PASS: TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_in_the_middle_of_the_last_24h_period#01 (0.67s) --- PASS: TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_in_the_middle_of_the_previous_24h_period#01 (3.42s) --- PASS: TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_exactly_at_the_beginning_of_this_24h_period#01 (2.82s) === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min/max_query_time_has_not_been_set_yet === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min/max_query_time_is_older_than_max_age === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_query_time_=_max_query_time === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_and_max_query_time_are_within_the_last_1h === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_and_max_query_time_are_within_the_last_2h === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_query_time_is_older_than_24h === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/max_query_time_is_older_than_24h_but_more_recent_than_max_query_age === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_query_time_is_older_than_24h_but_max_query_age_is_only_10m --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges (0.01s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min/max_query_time_has_not_been_set_yet (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min/max_query_time_is_older_than_max_age (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_query_time_=_max_query_time (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_and_max_query_time_are_within_the_last_1h (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_and_max_query_time_are_within_the_last_2h (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_query_time_is_older_than_24h (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/max_query_time_is_older_than_24h_but_more_recent_than_max_query_age (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_query_time_is_older_than_24h_but_max_query_age_is_only_10m (0.00s) PASS ok github.com/grafana/mimir/pkg/continuoustest 68.951s === RUN TestConfig_Validate === RUN TestConfig_Validate/default_config_should_pass === RUN TestConfig_Validate/should_fail_if_the_default_shard_size_is_negative === RUN TestConfig_Validate/should_pass_if_the_default_shard_size_>=_0 --- PASS: TestConfig_Validate (0.02s) --- PASS: TestConfig_Validate/default_config_should_pass (0.00s) --- PASS: TestConfig_Validate/should_fail_if_the_default_shard_size_is_negative (0.00s) --- PASS: TestConfig_Validate/should_pass_if_the_default_shard_size_>=_0 (0.00s) === RUN TestDistributor_Push === RUN TestDistributor_Push/A_push_to_3_happy_ingesters_should_succeed === RUN TestDistributor_Push/A_push_to_1_happy_ingesters_should_fail === RUN TestDistributor_Push/A_push_to_0_happy_ingesters_should_fail === RUN TestDistributor_Push/A_push_exceeding_burst_size_should_fail === RUN TestDistributor_Push/A_push_to_ingesters_with_a_current_sample_should_report_the_correct_metrics_with_no_metadata === RUN TestDistributor_Push/A_push_to_ingesters_without_samples_should_report_the_correct_metrics === RUN TestDistributor_Push/A_push_of_no_samples_shouldn't_block_or_return_error,_even_if_ingesters_are_sad === RUN TestDistributor_Push/A_push_to_2_happy_ingesters_should_succeed === RUN TestDistributor_Push/A_push_to_ingesters_with_an_old_sample_should_report_the_correct_metrics_with_no_metadata === RUN TestDistributor_Push/A_timed_out_push_should_fail --- PASS: TestDistributor_Push (1.57s) --- PASS: TestDistributor_Push/A_push_to_3_happy_ingesters_should_succeed (0.22s) --- PASS: TestDistributor_Push/A_push_to_1_happy_ingesters_should_fail (0.13s) --- PASS: TestDistributor_Push/A_push_to_0_happy_ingesters_should_fail (0.12s) --- PASS: TestDistributor_Push/A_push_exceeding_burst_size_should_fail (0.12s) --- PASS: TestDistributor_Push/A_push_to_ingesters_with_a_current_sample_should_report_the_correct_metrics_with_no_metadata (0.21s) --- PASS: TestDistributor_Push/A_push_to_ingesters_without_samples_should_report_the_correct_metrics (0.13s) --- PASS: TestDistributor_Push/A_push_of_no_samples_shouldn't_block_or_return_error,_even_if_ingesters_are_sad (0.13s) --- PASS: TestDistributor_Push/A_push_to_2_happy_ingesters_should_succeed (0.16s) --- PASS: TestDistributor_Push/A_push_to_ingesters_with_an_old_sample_should_report_the_correct_metrics_with_no_metadata (0.18s) --- PASS: TestDistributor_Push/A_timed_out_push_should_fail (0.15s) === RUN TestDistributor_ContextCanceledRequest --- PASS: TestDistributor_ContextCanceledRequest (0.13s) === RUN TestDistributor_MetricsCleanup --- PASS: TestDistributor_MetricsCleanup (0.19s) === RUN TestDistributor_PushRequestRateLimiter === RUN TestDistributor_PushRequestRateLimiter/request_limit_should_be_evenly_shared_across_distributors === RUN TestDistributor_PushRequestRateLimiter/request_limit_is_disabled_when_set_to_0 === RUN TestDistributor_PushRequestRateLimiter/request_burst_should_set_to_each_distributor --- PASS: TestDistributor_PushRequestRateLimiter (0.41s) --- PASS: TestDistributor_PushRequestRateLimiter/request_limit_should_be_evenly_shared_across_distributors (0.13s) --- PASS: TestDistributor_PushRequestRateLimiter/request_limit_is_disabled_when_set_to_0 (0.13s) --- PASS: TestDistributor_PushRequestRateLimiter/request_burst_should_set_to_each_distributor (0.13s) === RUN TestDistributor_PushIngestionRateLimiter === RUN TestDistributor_PushIngestionRateLimiter/evenly_share_the_ingestion_limit_across_distributors === RUN TestDistributor_PushIngestionRateLimiter/for_each_distributor,_set_an_ingestion_burst_limit. --- PASS: TestDistributor_PushIngestionRateLimiter (0.39s) --- PASS: TestDistributor_PushIngestionRateLimiter/evenly_share_the_ingestion_limit_across_distributors (0.26s) --- PASS: TestDistributor_PushIngestionRateLimiter/for_each_distributor,_set_an_ingestion_burst_limit. (0.12s) === RUN TestDistributor_PushInstanceLimits === RUN TestDistributor_PushInstanceLimits/below_rate_limit_on_first_request,_but_hits_the_rate_limit_afterwards distributor_test.go:793: Error Trace: /build/mimir/src/mimir/pkg/distributor/distributor_test.go:793 Error: Not equal: expected: *errors.fundamental(the write request has been rejected because the distributor exceeded the ingestion rate limit (err-mimir-distributor-max-ingestion-rate). To adjust the related limit, configure -distributor.instance-limits.max-ingestion-rate, or contact your service administrator.) actual : () Test: TestDistributor_PushInstanceLimits/below_rate_limit_on_first_request,_but_hits_the_rate_limit_afterwards distributor_test.go:791: Error Trace: /build/mimir/src/mimir/pkg/distributor/distributor_test.go:791 Error: Expected nil, but got: the write request has been rejected because the distributor exceeded the ingestion rate limit (err-mimir-distributor-max-ingestion-rate). To adjust the related limit, configure -distributor.instance-limits.max-ingestion-rate, or contact your service administrator. Test: TestDistributor_PushInstanceLimits/below_rate_limit_on_first_request,_but_hits_the_rate_limit_afterwards === RUN TestDistributor_PushInstanceLimits/below_inflight_size_limit === RUN TestDistributor_PushInstanceLimits/hits_inflight_size_limit === RUN TestDistributor_PushInstanceLimits/no_limits_limit === RUN TestDistributor_PushInstanceLimits/below_inflight_limit === RUN TestDistributor_PushInstanceLimits/hits_inflight_limit === RUN TestDistributor_PushInstanceLimits/below_ingestion_rate_limit === RUN TestDistributor_PushInstanceLimits/hits_rate_limit_on_first_request,_but_second_request_can_proceed --- FAIL: TestDistributor_PushInstanceLimits (2.89s) --- FAIL: TestDistributor_PushInstanceLimits/below_rate_limit_on_first_request,_but_hits_the_rate_limit_afterwards (1.57s) --- PASS: TestDistributor_PushInstanceLimits/below_inflight_size_limit (0.16s) --- PASS: TestDistributor_PushInstanceLimits/hits_inflight_size_limit (0.20s) --- PASS: TestDistributor_PushInstanceLimits/no_limits_limit (0.18s) --- PASS: TestDistributor_PushInstanceLimits/below_inflight_limit (0.25s) --- PASS: TestDistributor_PushInstanceLimits/hits_inflight_limit (0.12s) --- PASS: TestDistributor_PushInstanceLimits/below_ingestion_rate_limit (0.17s) --- PASS: TestDistributor_PushInstanceLimits/hits_rate_limit_on_first_request,_but_second_request_can_proceed (0.22s) === RUN TestDistributor_PushHAInstances === RUN TestDistributor_PushHAInstances/0 === RUN TestDistributor_PushHAInstances/1 === RUN TestDistributor_PushHAInstances/2 === RUN TestDistributor_PushHAInstances/3 --- PASS: TestDistributor_PushHAInstances (2.44s) --- PASS: TestDistributor_PushHAInstances/0 (2.01s) --- PASS: TestDistributor_PushHAInstances/1 (0.12s) --- PASS: TestDistributor_PushHAInstances/2 (0.19s) --- PASS: TestDistributor_PushHAInstances/3 (0.12s) === RUN TestDistributor_PushQuery === RUN TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=1) --- PASS: TestDistributor_PushQuery (0.35s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=7) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=6) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=4) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=2) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=9) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=9) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=8) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=7) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=6) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=5) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=4) (0.23s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=3) (0.18s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=2) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=1) (0.18s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) (0.21s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=9) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=7) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=8) (0.22s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=6) (0.19s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=5) (0.22s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=4) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=3) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=2) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) (0.17s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) (0.17s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=5)) (0.17s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=0)) (0.17s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=5)) (0.17s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=5)) (0.17s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=5)) (0.15s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=9) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=8) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=7) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=4) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=2) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=3) (0.21s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=9) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=8) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=7) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=5) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=4) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=2) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) (0.20s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=8) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=9) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=8) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=7) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=5) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=6) (0.21s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=3) (0.25s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=4) (0.35s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=2) (0.18s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=1) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) (0.21s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=9) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=7) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=6) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=4) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=3) (0.20s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=2) (0.23s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=1) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=0) (0.20s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) (0.16s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) (0.18s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=0)) (0.20s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=0)) (0.15s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=9) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=8) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=7) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=5) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=4) (0.18s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=3) (0.18s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=2) (0.21s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=0) (0.19s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) (0.18s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=7) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=6) (0.21s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=5) (0.22s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=4) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=3) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=2) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=9) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=8) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=7) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=5) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=4) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=2) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=3) (0.23s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=7) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) (0.21s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=9) (0.19s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=8) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=7) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=6) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=5) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=4) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=3) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=2) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=5)) (0.18s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=0)) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=8) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=9) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=6) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=7) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=4) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=5) (0.22s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=2) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=1) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=0) (0.18s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) (0.18s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) (0.22s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=9) (0.19s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=8) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=7) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=5) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=6) (0.19s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=4) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=2) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) (0.15s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=6) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=5) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=4) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=3) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=2) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=1) (0.16s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=0) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=9) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=9) (0.18s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=7) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=6) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=5) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=4) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=5) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=3) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=1) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=0) (0.23s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=9) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=8) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=7) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=6) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=5) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=4) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=3) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=2) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=1) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) (0.19s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=9) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=8) (0.20s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=7) (0.23s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=6) (0.22s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=5) (0.23s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=4) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=3) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=0)) (0.17s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=0)) (0.15s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=9) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=8) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=7) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=5) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=6) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=4) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=3) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=2) (0.16s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=0) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=4) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=8) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=7) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=6) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=4) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=3) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=2) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=9) (0.17s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) (0.18s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=8) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=7) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=6) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=5) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=4) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=3) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=2) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=1) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=0) (0.16s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=9) (0.20s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) (0.24s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=8) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=7) (0.18s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=5) (0.20s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=6) (0.24s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=3) (0.21s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=4) (0.21s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=1) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=2) (0.20s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=0) (0.14s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) (0.15s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=0)) (0.16s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=0)) (0.16s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=9) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=8) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=8) (0.20s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=7) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=6) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=3) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=2) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=1) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) (0.24s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=9) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=8) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=7) (0.19s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=5) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=4) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=4) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=3) (0.18s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=1) (0.20s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=0) (0.15s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) (0.18s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=9) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=8) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=7) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=6) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=3) (0.20s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=2) (0.20s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=4) (0.21s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=0) (0.21s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) (0.22s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=8) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=7) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=6) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=5) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=2) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=1) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) (0.17s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=0)) (0.16s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=5)) (0.21s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=5)) (0.20s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=0)) (0.16s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=5)) (0.15s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=5)) (0.19s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=0)) (0.19s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=9) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=8) (0.22s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=7) (0.22s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=6) (0.22s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=5) (0.21s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=4) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=8) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=3) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=2) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=1) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=9) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=7) (0.18s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=6) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=5) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=4) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=2) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=1) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=0) (0.14s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=9) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=8) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=7) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=5) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=4) (0.24s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=2) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=1) (0.20s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) (0.21s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=9) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=7) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=8) (0.18s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=5) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=6) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=4) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=3) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=0) (0.16s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) (0.16s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=5)) (0.15s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=0)) (0.15s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=5)) (0.15s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=5)) (0.17s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=0)) (0.20s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=8) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=7) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=5) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=6) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=3) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=4) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=2) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=1) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=0) (0.18s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) (0.18s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) (0.16s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=9) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=1) (0.13s) === RUN TestDistributor_QueryStream_ShouldReturnErrorIfMaxChunksPerQueryLimitIsReached --- PASS: TestDistributor_QueryStream_ShouldReturnErrorIfMaxChunksPerQueryLimitIsReached (0.16s) === RUN TestDistributor_QueryStream_ShouldReturnErrorIfMaxSeriesPerQueryLimitIsReached --- PASS: TestDistributor_QueryStream_ShouldReturnErrorIfMaxSeriesPerQueryLimitIsReached (0.15s) === RUN TestDistributor_QueryStream_ShouldReturnErrorIfMaxChunkBytesPerQueryLimitIsReached --- PASS: TestDistributor_QueryStream_ShouldReturnErrorIfMaxChunkBytesPerQueryLimitIsReached (0.19s) === RUN TestDistributor_Push_LabelRemoval --- PASS: TestDistributor_Push_LabelRemoval (0.90s) === RUN TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime === RUN TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_1_and_dropped_label_due_to_config === RUN TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_1_and_dropped_HA_replica_label === RUN TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_2_with_value_1 === RUN TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_2 === RUN TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_1 --- PASS: TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime (0.62s) --- PASS: TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_1_and_dropped_label_due_to_config (0.12s) --- PASS: TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_1_and_dropped_HA_replica_label (0.12s) --- PASS: TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_2_with_value_1 (0.13s) --- PASS: TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_2 (0.12s) --- PASS: TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_1 (0.12s) === RUN TestDistributor_Push_LabelNameValidation === RUN TestDistributor_Push_LabelNameValidation/label_name_validation_is_on_by_default === RUN TestDistributor_Push_LabelNameValidation/label_name_validation_can_be_skipped_via_config === RUN TestDistributor_Push_LabelNameValidation/label_name_validation_can_be_skipped_via_WriteRequest_parameter --- PASS: TestDistributor_Push_LabelNameValidation (0.37s) --- PASS: TestDistributor_Push_LabelNameValidation/label_name_validation_is_on_by_default (0.13s) --- PASS: TestDistributor_Push_LabelNameValidation/label_name_validation_can_be_skipped_via_config (0.12s) --- PASS: TestDistributor_Push_LabelNameValidation/label_name_validation_can_be_skipped_via_WriteRequest_parameter (0.12s) === RUN TestDistributor_Push_ExemplarValidation === RUN TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_empty_series_label_name === RUN TestDistributor_Push_ExemplarValidation/valid_exemplar === RUN TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_no_labels === RUN TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_no_timestamp === RUN TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_too_long_labelset === RUN TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_too_many_series_labels === RUN TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_duplicate_series_labels --- PASS: TestDistributor_Push_ExemplarValidation (0.94s) --- PASS: TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_empty_series_label_name (0.12s) --- PASS: TestDistributor_Push_ExemplarValidation/valid_exemplar (0.13s) --- PASS: TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_no_labels (0.15s) --- PASS: TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_no_timestamp (0.13s) --- PASS: TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_too_long_labelset (0.15s) --- PASS: TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_too_many_series_labels (0.13s) --- PASS: TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_duplicate_series_labels (0.12s) === RUN TestDistributor_Push_HistogramValidation === RUN TestDistributor_Push_HistogramValidation/valid_histogram === RUN TestDistributor_Push_HistogramValidation/too_new_histogram === RUN TestDistributor_Push_HistogramValidation/valid_float_histogram === RUN TestDistributor_Push_HistogramValidation/too_new_float_histogram --- PASS: TestDistributor_Push_HistogramValidation (0.52s) --- PASS: TestDistributor_Push_HistogramValidation/valid_histogram (0.12s) --- PASS: TestDistributor_Push_HistogramValidation/too_new_histogram (0.12s) --- PASS: TestDistributor_Push_HistogramValidation/valid_float_histogram (0.12s) --- PASS: TestDistributor_Push_HistogramValidation/too_new_float_histogram (0.15s) === RUN TestDistributor_ExemplarValidation === RUN TestDistributor_ExemplarValidation/one_old,_one_new,_separate_series === RUN TestDistributor_ExemplarValidation/multi_exemplars === RUN TestDistributor_ExemplarValidation/one_old,_one_new,_same_series === RUN TestDistributor_ExemplarValidation/disable_exemplars === RUN TestDistributor_ExemplarValidation/valid_exemplars --- PASS: TestDistributor_ExemplarValidation (0.71s) --- PASS: TestDistributor_ExemplarValidation/one_old,_one_new,_separate_series (0.12s) --- PASS: TestDistributor_ExemplarValidation/multi_exemplars (0.12s) --- PASS: TestDistributor_ExemplarValidation/one_old,_one_new,_same_series (0.12s) --- PASS: TestDistributor_ExemplarValidation/disable_exemplars (0.15s) --- PASS: TestDistributor_ExemplarValidation/valid_exemplars (0.19s) === RUN TestSlowQueries === RUN TestSlowQueries/0 === RUN TestSlowQueries/1 === RUN TestSlowQueries/2 === RUN TestSlowQueries/3 --- PASS: TestSlowQueries (0.92s) --- PASS: TestSlowQueries/0 (0.24s) --- PASS: TestSlowQueries/1 (0.22s) --- PASS: TestSlowQueries/2 (0.22s) --- PASS: TestSlowQueries/3 (0.24s) === RUN TestDistributor_MetricsForLabelMatchers === RUN TestDistributor_MetricsForLabelMatchers/should_return_an_empty_response_if_no_metric_match === RUN TestDistributor_MetricsForLabelMatchers/should_filter_metrics_by_single_matcher === RUN TestDistributor_MetricsForLabelMatchers/should_filter_metrics_by_multiple_matchers === RUN TestDistributor_MetricsForLabelMatchers/should_return_all_matching_metrics_even_if_their_FastFingerprint_collide === RUN TestDistributor_MetricsForLabelMatchers/should_query_only_ingesters_belonging_to_tenant's_subring_if_shuffle_shard_size_is_set --- PASS: TestDistributor_MetricsForLabelMatchers (0.63s) --- PASS: TestDistributor_MetricsForLabelMatchers/should_return_an_empty_response_if_no_metric_match (0.14s) --- PASS: TestDistributor_MetricsForLabelMatchers/should_filter_metrics_by_single_matcher (0.13s) --- PASS: TestDistributor_MetricsForLabelMatchers/should_filter_metrics_by_multiple_matchers (0.12s) --- PASS: TestDistributor_MetricsForLabelMatchers/should_return_all_matching_metrics_even_if_their_FastFingerprint_collide (0.12s) --- PASS: TestDistributor_MetricsForLabelMatchers/should_query_only_ingesters_belonging_to_tenant's_subring_if_shuffle_shard_size_is_set (0.12s) === RUN TestDistributor_LabelNames === RUN TestDistributor_LabelNames/should_return_an_empty_response_if_no_metric_match === RUN TestDistributor_LabelNames/should_filter_metrics_by_single_matcher === RUN TestDistributor_LabelNames/should_filter_metrics_by_multiple_matchers === RUN TestDistributor_LabelNames/should_query_only_ingesters_belonging_to_tenant's_subring_if_shuffle_sharding_is_enabled --- PASS: TestDistributor_LabelNames (0.51s) --- PASS: TestDistributor_LabelNames/should_return_an_empty_response_if_no_metric_match (0.13s) --- PASS: TestDistributor_LabelNames/should_filter_metrics_by_single_matcher (0.13s) --- PASS: TestDistributor_LabelNames/should_filter_metrics_by_multiple_matchers (0.12s) --- PASS: TestDistributor_LabelNames/should_query_only_ingesters_belonging_to_tenant's_subring_if_shuffle_sharding_is_enabled (0.13s) === RUN TestDistributor_MetricsMetadata === RUN TestDistributor_MetricsMetadata/should_query_only_ingesters_belonging_to_tenant's_subring_if_shuffle_sharding_is_enabled === RUN TestDistributor_MetricsMetadata/should_query_all_ingesters_if_shuffle_sharding_is_enabled_but_shard_size_is_0 --- PASS: TestDistributor_MetricsMetadata (0.24s) --- PASS: TestDistributor_MetricsMetadata/should_query_only_ingesters_belonging_to_tenant's_subring_if_shuffle_sharding_is_enabled (0.12s) --- PASS: TestDistributor_MetricsMetadata/should_query_all_ingesters_if_shuffle_sharding_is_enabled_but_shard_size_is_0 (0.12s) === RUN TestDistributor_LabelNamesAndValuesLimitTest === RUN TestDistributor_LabelNamesAndValuesLimitTest/expected_error_if_sizeLimit_is_reached === RUN TestDistributor_LabelNamesAndValuesLimitTest/expected_no_error_if_sizeLimit_is_not_reached --- PASS: TestDistributor_LabelNamesAndValuesLimitTest (0.25s) --- PASS: TestDistributor_LabelNamesAndValuesLimitTest/expected_error_if_sizeLimit_is_reached (0.12s) --- PASS: TestDistributor_LabelNamesAndValuesLimitTest/expected_no_error_if_sizeLimit_is_not_reached (0.12s) === RUN TestDistributor_LabelNamesAndValues === RUN TestDistributor_LabelNamesAndValues/should_group_values_of_labels_by_label_name_and_return_only_distinct_label_values === RUN TestDistributor_LabelNamesAndValues/should_return_the_results_if_zone_awareness_is_enabled_and_only_2_zones_return_the_results --- PASS: TestDistributor_LabelNamesAndValues (0.25s) --- PASS: TestDistributor_LabelNamesAndValues/should_group_values_of_labels_by_label_name_and_return_only_distinct_label_values (0.12s) --- PASS: TestDistributor_LabelNamesAndValues/should_return_the_results_if_zone_awareness_is_enabled_and_only_2_zones_return_the_results (0.12s) === RUN TestDistributor_LabelValuesCardinality_ExpectedAllIngestersResponsesToBeCompleted --- PASS: TestDistributor_LabelValuesCardinality_ExpectedAllIngestersResponsesToBeCompleted (15.09s) === RUN TestDistributor_LabelNamesAndValues_ExpectedAllPossibleLabelNamesAndValuesToBeReturned --- PASS: TestDistributor_LabelNamesAndValues_ExpectedAllPossibleLabelNamesAndValuesToBeReturned (14.28s) === RUN TestDistributor_IngestionIsControlledByForwarder === RUN TestDistributor_IngestionIsControlledByForwarder/do_ingest_with_only_samples === RUN TestDistributor_IngestionIsControlledByForwarder/don't_ingest_with_only_samples === RUN TestDistributor_IngestionIsControlledByForwarder/do_ingest_with_metadata === RUN TestDistributor_IngestionIsControlledByForwarder/don't_ingest_with_metadata === RUN TestDistributor_IngestionIsControlledByForwarder/do_ingest_with_exemplars === RUN TestDistributor_IngestionIsControlledByForwarder/don't_ingest_with_exemplars === RUN TestDistributor_IngestionIsControlledByForwarder/do_ingest_with_histograms === RUN TestDistributor_IngestionIsControlledByForwarder/don't_ingest_with_histograms --- PASS: TestDistributor_IngestionIsControlledByForwarder (1.19s) --- PASS: TestDistributor_IngestionIsControlledByForwarder/do_ingest_with_only_samples (0.14s) --- PASS: TestDistributor_IngestionIsControlledByForwarder/don't_ingest_with_only_samples (0.14s) --- PASS: TestDistributor_IngestionIsControlledByForwarder/do_ingest_with_metadata (0.13s) --- PASS: TestDistributor_IngestionIsControlledByForwarder/don't_ingest_with_metadata (0.14s) --- PASS: TestDistributor_IngestionIsControlledByForwarder/do_ingest_with_exemplars (0.13s) --- PASS: TestDistributor_IngestionIsControlledByForwarder/don't_ingest_with_exemplars (0.15s) --- PASS: TestDistributor_IngestionIsControlledByForwarder/do_ingest_with_histograms (0.14s) --- PASS: TestDistributor_IngestionIsControlledByForwarder/don't_ingest_with_histograms (0.20s) === RUN TestDistributor_LabelValuesCardinality === RUN TestDistributor_LabelValuesCardinality/should_return_an_empty_map_if_no_label_names === RUN TestDistributor_LabelValuesCardinality/should_return_a_map_with_the_label_values_and_series_occurrences_of_a_single_label_name === RUN TestDistributor_LabelValuesCardinality/should_return_a_map_with_the_label_values_and_series_occurrences_of_all_the_label_names --- PASS: TestDistributor_LabelValuesCardinality (0.44s) --- PASS: TestDistributor_LabelValuesCardinality/should_return_an_empty_map_if_no_label_names (0.15s) --- PASS: TestDistributor_LabelValuesCardinality/should_return_a_map_with_the_label_values_and_series_occurrences_of_a_single_label_name (0.13s) --- PASS: TestDistributor_LabelValuesCardinality/should_return_a_map_with_the_label_values_and_series_occurrences_of_all_the_label_names (0.16s) === RUN TestDistributor_LabelValuesCardinalityLimit === RUN TestDistributor_LabelValuesCardinalityLimit/should_return_a_httpgrpc_error_if_the_maximum_number_of_label_names_per_request_is_reached === RUN TestDistributor_LabelValuesCardinalityLimit/should_succeed_if_the_maximum_number_of_label_names_per_request_is_not_reached --- PASS: TestDistributor_LabelValuesCardinalityLimit (0.32s) --- PASS: TestDistributor_LabelValuesCardinalityLimit/should_return_a_httpgrpc_error_if_the_maximum_number_of_label_names_per_request_is_reached (0.13s) --- PASS: TestDistributor_LabelValuesCardinalityLimit/should_succeed_if_the_maximum_number_of_label_names_per_request_is_not_reached (0.19s) === RUN TestDistributor_LabelValuesCardinality_Concurrency === RUN TestDistributor_LabelValuesCardinality_Concurrency/should_fail_with_an_error_if_at_least_one_ingester's_LabelValuesCardinality_and/or_UserStats_operations_fails --- PASS: TestDistributor_LabelValuesCardinality_Concurrency (0.12s) --- PASS: TestDistributor_LabelValuesCardinality_Concurrency/should_fail_with_an_error_if_at_least_one_ingester's_LabelValuesCardinality_and/or_UserStats_operations_fails (0.12s) === RUN TestHaDedupeMiddleware === RUN TestHaDedupeMiddleware/no_changes_on_empty_request === RUN TestHaDedupeMiddleware/no_changes_if_accept_HA_samples_is_false === RUN TestHaDedupeMiddleware/remove_replica_label_with_HA_tracker_disabled === RUN TestHaDedupeMiddleware/do_nothing_without_user_in_context,_don't_even_call_next === RUN TestHaDedupeMiddleware/perform_HA_deduplication === RUN TestHaDedupeMiddleware/exceed_max_ha_clusters_limit --- PASS: TestHaDedupeMiddleware (7.29s) --- PASS: TestHaDedupeMiddleware/no_changes_on_empty_request (2.01s) --- PASS: TestHaDedupeMiddleware/no_changes_if_accept_HA_samples_is_false (2.01s) --- PASS: TestHaDedupeMiddleware/remove_replica_label_with_HA_tracker_disabled (0.12s) --- PASS: TestHaDedupeMiddleware/do_nothing_without_user_in_context,_don't_even_call_next (1.01s) --- PASS: TestHaDedupeMiddleware/perform_HA_deduplication (2.01s) --- PASS: TestHaDedupeMiddleware/exceed_max_ha_clusters_limit (0.13s) === RUN TestInstanceLimitsBeforeHaDedupe --- PASS: TestInstanceLimitsBeforeHaDedupe (2.01s) === RUN TestRelabelMiddleware === RUN TestRelabelMiddleware/do_nothing === RUN TestRelabelMiddleware/no_user_in_context === RUN TestRelabelMiddleware/apply_a_relabel_rule === RUN TestRelabelMiddleware/drop_two_out_of_three_labels === RUN TestRelabelMiddleware/drop_entire_series_if_they_have_no_labels --- PASS: TestRelabelMiddleware (0.63s) --- PASS: TestRelabelMiddleware/do_nothing (0.14s) --- PASS: TestRelabelMiddleware/no_user_in_context (0.13s) --- PASS: TestRelabelMiddleware/apply_a_relabel_rule (0.12s) --- PASS: TestRelabelMiddleware/drop_two_out_of_three_labels (0.12s) --- PASS: TestRelabelMiddleware/drop_entire_series_if_they_have_no_labels (0.12s) === RUN TestHaDedupeAndRelabelBeforeForwarding --- PASS: TestHaDedupeAndRelabelBeforeForwarding (0.13s) === RUN TestValidationBeforeForwarding --- PASS: TestValidationBeforeForwarding (2.01s) === RUN TestDistributorValidation === RUN TestDistributorValidation/0 === RUN TestDistributorValidation/1 === RUN TestDistributorValidation/2 === RUN TestDistributorValidation/3 === RUN TestDistributorValidation/4 === RUN TestDistributorValidation/5 === RUN TestDistributorValidation/6 --- PASS: TestDistributorValidation (0.98s) --- PASS: TestDistributorValidation/0 (0.12s) --- PASS: TestDistributorValidation/1 (0.18s) --- PASS: TestDistributorValidation/2 (0.13s) --- PASS: TestDistributorValidation/3 (0.12s) --- PASS: TestDistributorValidation/4 (0.14s) --- PASS: TestDistributorValidation/5 (0.16s) --- PASS: TestDistributorValidation/6 (0.12s) === RUN TestRemoveReplicaLabel --- PASS: TestRemoveReplicaLabel (0.00s) === RUN TestShardByAllLabelsReturnsWrongResultsForUnsortedLabels --- PASS: TestShardByAllLabelsReturnsWrongResultsForUnsortedLabels (0.00s) === RUN TestSortLabels --- PASS: TestSortLabels (0.00s) === RUN TestDistributor_Push_Relabel --- PASS: TestDistributor_Push_Relabel (0.38s) === RUN TestDistributor_MetricsWithRequestModifications === RUN TestDistributor_MetricsWithRequestModifications/No_modifications === RUN TestDistributor_MetricsWithRequestModifications/Drop_samples_via_relabel_rules === RUN TestDistributor_MetricsWithRequestModifications/Drop_samples_via_drop_label_rule === RUN TestDistributor_MetricsWithRequestModifications/Drop_requests_via_ha-dedupe === RUN TestDistributor_MetricsWithRequestModifications/Some_samples_get_forwarded === RUN TestDistributor_MetricsWithRequestModifications/Drop_examplars_via_limit === RUN TestDistributor_MetricsWithRequestModifications/Drop_half_of_samples_via_label_value_length_limit === RUN TestDistributor_MetricsWithRequestModifications/Drop_metadata_via_label_value_length_limit,_except_help --- PASS: TestDistributor_MetricsWithRequestModifications (2.50s) --- PASS: TestDistributor_MetricsWithRequestModifications/No_modifications (0.15s) --- PASS: TestDistributor_MetricsWithRequestModifications/Drop_samples_via_relabel_rules (0.18s) --- PASS: TestDistributor_MetricsWithRequestModifications/Drop_samples_via_drop_label_rule (0.13s) --- PASS: TestDistributor_MetricsWithRequestModifications/Drop_requests_via_ha-dedupe (0.17s) --- PASS: TestDistributor_MetricsWithRequestModifications/Some_samples_get_forwarded (0.21s) --- PASS: TestDistributor_MetricsWithRequestModifications/Drop_examplars_via_limit (0.16s) --- PASS: TestDistributor_MetricsWithRequestModifications/Drop_half_of_samples_via_label_value_length_limit (0.13s) --- PASS: TestDistributor_MetricsWithRequestModifications/Drop_metadata_via_label_value_length_limit,_except_help (0.13s) === RUN TestDistributor_CleanupIsDoneAfterLastIngesterReturns --- PASS: TestDistributor_CleanupIsDoneAfterLastIngesterReturns (0.12s) === RUN TestSeriesAreShardedToCorrectIngesters --- PASS: TestSeriesAreShardedToCorrectIngesters (0.27s) === RUN TestHATrackerConfig_Validate === PAUSE TestHATrackerConfig_Validate === RUN TestWatchPrefixAssignment --- PASS: TestWatchPrefixAssignment (0.00s) === RUN TestCheckReplicaOverwriteTimeout --- PASS: TestCheckReplicaOverwriteTimeout (1.00s) === RUN TestCheckReplicaMultiCluster --- PASS: TestCheckReplicaMultiCluster (2.00s) === RUN TestCheckReplicaMultiClusterTimeout --- PASS: TestCheckReplicaMultiClusterTimeout (2.00s) === RUN TestCheckReplicaUpdateTimeout --- PASS: TestCheckReplicaUpdateTimeout (2.00s) === RUN TestCheckReplicaMultiUser --- PASS: TestCheckReplicaMultiUser (2.00s) === RUN TestCheckReplicaUpdateTimeoutJitter === PAUSE TestCheckReplicaUpdateTimeoutJitter === RUN TestFindHALabels --- PASS: TestFindHALabels (0.00s) === RUN TestHATrackerConfig_ShouldCustomizePrefixDefaultValue --- PASS: TestHATrackerConfig_ShouldCustomizePrefixDefaultValue (0.00s) === RUN TestHAClustersLimit --- PASS: TestHAClustersLimit (0.05s) === RUN TestTooManyClustersError --- PASS: TestTooManyClustersError (0.00s) === RUN TestReplicasNotMatchError --- PASS: TestReplicasNotMatchError (0.00s) === RUN TestHATracker_MetricsCleanup --- PASS: TestHATracker_MetricsCleanup (0.01s) === RUN TestCheckReplicaCleanup --- PASS: TestCheckReplicaCleanup (2.00s) === RUN TestHealthyInstanceDelegate_OnRingInstanceHeartbeat === RUN TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/all_instances_healthy_and_active === RUN TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/all_instances_healthy_not_all_instances_active === RUN TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/some_instances_healthy_all_instances_active === RUN TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/some_instances_healthy_but_timeout_disabled_all_instances_active --- PASS: TestHealthyInstanceDelegate_OnRingInstanceHeartbeat (0.00s) --- PASS: TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/all_instances_healthy_and_active (0.00s) --- PASS: TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/all_instances_healthy_not_all_instances_active (0.00s) --- PASS: TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/some_instances_healthy_all_instances_active (0.00s) --- PASS: TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/some_instances_healthy_but_timeout_disabled_all_instances_active (0.00s) === RUN TestInstanceLimitsUnmarshal --- PASS: TestInstanceLimitsUnmarshal (0.00s) === RUN TestMergeSamplesIntoFirstDuplicates --- PASS: TestMergeSamplesIntoFirstDuplicates (0.00s) === RUN TestMergeSamplesIntoFirst --- PASS: TestMergeSamplesIntoFirst (0.00s) === RUN TestMergeSamplesIntoFirstNilA --- PASS: TestMergeSamplesIntoFirstNilA (0.00s) === RUN TestMergeSamplesIntoFirstNilB --- PASS: TestMergeSamplesIntoFirstNilB (0.00s) === RUN TestMergeExemplars === RUN TestMergeExemplars/test0 === RUN TestMergeExemplars/test1 === RUN TestMergeExemplars/test2 === RUN TestMergeExemplars/test3 === RUN TestMergeExemplars/test4 === RUN TestMergeExemplars/test5 === RUN TestMergeExemplars/test6 --- PASS: TestMergeExemplars (0.00s) --- PASS: TestMergeExemplars/test0 (0.00s) --- PASS: TestMergeExemplars/test1 (0.00s) --- PASS: TestMergeExemplars/test2 (0.00s) --- PASS: TestMergeExemplars/test3 (0.00s) --- PASS: TestMergeExemplars/test4 (0.00s) --- PASS: TestMergeExemplars/test5 (0.00s) --- PASS: TestMergeExemplars/test6 (0.00s) === RUN TestIngestionRateStrategy === RUN TestIngestionRateStrategy/rate_limiter_should_share_the_limit_across_the_number_of_distributors === RUN TestIngestionRateStrategy/infinite_rate_limiter_should_return_unlimited_settings --- PASS: TestIngestionRateStrategy (0.00s) --- PASS: TestIngestionRateStrategy/rate_limiter_should_share_the_limit_across_the_number_of_distributors (0.00s) --- PASS: TestIngestionRateStrategy/infinite_rate_limiter_should_return_unlimited_settings (0.00s) === CONT TestHATrackerConfig_Validate === CONT TestCheckReplicaUpdateTimeoutJitter === RUN TestCheckReplicaUpdateTimeoutJitter/should_refresh_the_replica_if_the_update_timeout_is_expired_(with_jitter) === RUN TestHATrackerConfig_Validate/should_pass_with_default_config === RUN TestHATrackerConfig_Validate/should_fail_if_max_update_timeout_jitter_is_negative === RUN TestHATrackerConfig_Validate/should_fail_if_failover_timeout_is_<_update_timeout_+_jitter_+_1_sec === RUN TestHATrackerConfig_Validate/should_pass_if_failover_timeout_is_>=_update_timeout_+_jitter_+_1_sec === RUN TestHATrackerConfig_Validate/should_fail_if_KV_backend_is_set_to_memberlist --- PASS: TestHATrackerConfig_Validate (0.02s) --- PASS: TestHATrackerConfig_Validate/should_pass_with_default_config (0.00s) --- PASS: TestHATrackerConfig_Validate/should_fail_if_max_update_timeout_jitter_is_negative (0.00s) --- PASS: TestHATrackerConfig_Validate/should_fail_if_failover_timeout_is_<_update_timeout_+_jitter_+_1_sec (0.00s) --- PASS: TestHATrackerConfig_Validate/should_pass_if_failover_timeout_is_>=_update_timeout_+_jitter_+_1_sec (0.00s) --- PASS: TestHATrackerConfig_Validate/should_fail_if_KV_backend_is_set_to_memberlist (0.00s) === RUN TestCheckReplicaUpdateTimeoutJitter/should_not_refresh_the_replica_if_the_update_timeout_is_not_expired_yet_(without_jitter) === RUN TestCheckReplicaUpdateTimeoutJitter/should_refresh_the_replica_if_the_update_timeout_is_expired_(without_jitter) === RUN TestCheckReplicaUpdateTimeoutJitter/should_not_refresh_the_replica_if_the_update_timeout_is_not_expired_yet_(with_jitter) --- PASS: TestCheckReplicaUpdateTimeoutJitter (3.02s) --- PASS: TestCheckReplicaUpdateTimeoutJitter/should_refresh_the_replica_if_the_update_timeout_is_expired_(with_jitter) (1.01s) --- PASS: TestCheckReplicaUpdateTimeoutJitter/should_not_refresh_the_replica_if_the_update_timeout_is_not_expired_yet_(without_jitter) (0.00s) --- PASS: TestCheckReplicaUpdateTimeoutJitter/should_refresh_the_replica_if_the_update_timeout_is_expired_(without_jitter) (2.00s) --- PASS: TestCheckReplicaUpdateTimeoutJitter/should_not_refresh_the_replica_if_the_update_timeout_is_not_expired_yet_(with_jitter) (0.00s) FAIL FAIL github.com/grafana/mimir/pkg/distributor 111.286s ? github.com/grafana/mimir/pkg/flusher [no test files] === RUN TestForwardingSamplesSuccessfullyToSingleTarget --- PASS: TestForwardingSamplesSuccessfullyToSingleTarget (0.02s) === RUN TestForwardingOmitOldSamples === RUN TestForwardingOmitOldSamples/drop_nothing === RUN TestForwardingOmitOldSamples/drop_some_of_the_samples,_ingest_true === RUN TestForwardingOmitOldSamples/drop_some_of_the_samples,_ingest_false === RUN TestForwardingOmitOldSamples/split_one_sample_slice_in_the_middle === RUN TestForwardingOmitOldSamples/drop_all_of_the_samples --- PASS: TestForwardingOmitOldSamples (0.04s) --- PASS: TestForwardingOmitOldSamples/drop_nothing (0.01s) --- PASS: TestForwardingOmitOldSamples/drop_some_of_the_samples,_ingest_true (0.01s) --- PASS: TestForwardingOmitOldSamples/drop_some_of_the_samples,_ingest_false (0.01s) --- PASS: TestForwardingOmitOldSamples/split_one_sample_slice_in_the_middle (0.01s) --- PASS: TestForwardingOmitOldSamples/drop_all_of_the_samples (0.00s) === RUN TestForwardingEnsureThatPooledObjectsGetReturned === RUN TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_one_metric_and_don't_ingest_them === RUN TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_one_metric_and_ingest_them === RUN TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_and_don't_ingest_them === RUN TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_to_and_ingest_them === RUN TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_to_and_ingest_half_of_them_(1) === RUN TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_and_ingest_half_of_them_(2) === RUN TestForwardingEnsureThatPooledObjectsGetReturned/don't_forward_samples,_just_ingest_them --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned (0.09s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_one_metric_and_don't_ingest_them (0.01s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_one_metric_and_ingest_them (0.01s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_and_don't_ingest_them (0.02s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_to_and_ingest_them (0.02s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_to_and_ingest_half_of_them_(1) (0.01s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_and_ingest_half_of_them_(2) (0.01s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/don't_forward_samples,_just_ingest_them (0.00s) === RUN TestDropSamplesBefore === RUN TestDropSamplesBefore/no_samples === RUN TestDropSamplesBefore/all_samples_before_cutoff === RUN TestDropSamplesBefore/all_samples_after_cutoff === RUN TestDropSamplesBefore/cut_some_samples_off === RUN TestDropSamplesBefore/unsorted_sample_slice --- PASS: TestDropSamplesBefore (0.00s) --- PASS: TestDropSamplesBefore/no_samples (0.00s) --- PASS: TestDropSamplesBefore/all_samples_before_cutoff (0.00s) --- PASS: TestDropSamplesBefore/all_samples_after_cutoff (0.00s) --- PASS: TestDropSamplesBefore/cut_some_samples_off (0.00s) --- PASS: TestDropSamplesBefore/unsorted_sample_slice (0.00s) === RUN TestForwardingToHTTPGrpcTarget --- PASS: TestForwardingToHTTPGrpcTarget (0.03s) === RUN TestUsingPools --- PASS: TestUsingPools (0.00s) PASS ok github.com/grafana/mimir/pkg/distributor/forwarding 0.471s === RUN TestFrontend_RequestHostHeaderWhenDownstreamURLIsConfigured --- PASS: TestFrontend_RequestHostHeaderWhenDownstreamURLIsConfigured (0.07s) === RUN TestFrontend_LogsSlowQueriesFormValues --- PASS: TestFrontend_LogsSlowQueriesFormValues (0.08s) === RUN TestFrontend_ReturnsRequestBodyTooLargeError --- PASS: TestFrontend_ReturnsRequestBodyTooLargeError (0.02s) PASS ok github.com/grafana/mimir/pkg/frontend 0.539s ? github.com/grafana/mimir/pkg/frontend/v1/frontendv1pb [no test files] ? github.com/grafana/mimir/pkg/frontend/v2/frontendv2pb [no test files] === RUN Test_cardinalityEstimateBucket_GenerateCacheKey_keyFormat === RUN Test_cardinalityEstimateBucket_GenerateCacheKey_keyFormat/instant_query === PAUSE Test_cardinalityEstimateBucket_GenerateCacheKey_keyFormat/instant_query === RUN Test_cardinalityEstimateBucket_GenerateCacheKey_keyFormat/range_query === PAUSE Test_cardinalityEstimateBucket_GenerateCacheKey_keyFormat/range_query === RUN Test_cardinalityEstimateBucket_GenerateCacheKey_keyFormat/range_query_with_large_range === PAUSE Test_cardinalityEstimateBucket_GenerateCacheKey_keyFormat/range_query_with_large_range === CONT Test_cardinalityEstimateBucket_GenerateCacheKey_keyFormat/instant_query === CONT Test_cardinalityEstimateBucket_GenerateCacheKey_keyFormat/range_query_with_large_range === CONT Test_cardinalityEstimateBucket_GenerateCacheKey_keyFormat/range_query --- PASS: Test_cardinalityEstimateBucket_GenerateCacheKey_keyFormat (0.00s) --- PASS: Test_cardinalityEstimateBucket_GenerateCacheKey_keyFormat/range_query_with_large_range (0.00s) --- PASS: Test_cardinalityEstimateBucket_GenerateCacheKey_keyFormat/instant_query (0.00s) --- PASS: Test_cardinalityEstimateBucket_GenerateCacheKey_keyFormat/range_query (0.00s) === RUN Test_cardinalityEstimation_lookupCardinalityForKey === RUN Test_cardinalityEstimation_lookupCardinalityForKey/nil_cache === RUN Test_cardinalityEstimation_lookupCardinalityForKey/cache_hit === RUN Test_cardinalityEstimation_lookupCardinalityForKey/cache_miss --- PASS: Test_cardinalityEstimation_lookupCardinalityForKey (0.00s) --- PASS: Test_cardinalityEstimation_lookupCardinalityForKey/nil_cache (0.00s) --- PASS: Test_cardinalityEstimation_lookupCardinalityForKey/cache_hit (0.00s) --- PASS: Test_cardinalityEstimation_lookupCardinalityForKey/cache_miss (0.00s) === RUN Test_cardinalityEstimation_Do === RUN Test_cardinalityEstimation_Do/no_tenantID === RUN Test_cardinalityEstimation_Do/downstream_error === RUN Test_cardinalityEstimation_Do/with_populated_cache_and_unchanged_cardinality === RUN Test_cardinalityEstimation_Do/with_populated_cache_and_marginally_changed_cardinality === RUN Test_cardinalityEstimation_Do/with_populated_cache_and_significantly_changed_cardinality === RUN Test_cardinalityEstimation_Do/with_empty_cache --- PASS: Test_cardinalityEstimation_Do (0.00s) --- PASS: Test_cardinalityEstimation_Do/no_tenantID (0.00s) --- PASS: Test_cardinalityEstimation_Do/downstream_error (0.00s) --- PASS: Test_cardinalityEstimation_Do/with_populated_cache_and_unchanged_cardinality (0.00s) --- PASS: Test_cardinalityEstimation_Do/with_populated_cache_and_marginally_changed_cardinality (0.00s) --- PASS: Test_cardinalityEstimation_Do/with_populated_cache_and_significantly_changed_cardinality (0.00s) --- PASS: Test_cardinalityEstimation_Do/with_empty_cache (0.00s) === RUN Test_cardinalityEstimateBucket_GenerateCacheKey_requestEquality === RUN Test_cardinalityEstimateBucket_GenerateCacheKey_requestEquality/same_tenant,_same_request === PAUSE Test_cardinalityEstimateBucket_GenerateCacheKey_requestEquality/same_tenant,_same_request === RUN Test_cardinalityEstimateBucket_GenerateCacheKey_requestEquality/different_tenant,_same_request === PAUSE Test_cardinalityEstimateBucket_GenerateCacheKey_requestEquality/different_tenant,_same_request === RUN Test_cardinalityEstimateBucket_GenerateCacheKey_requestEquality/same_tenant,_same_query_with_start_time_in_same_bucket === PAUSE Test_cardinalityEstimateBucket_GenerateCacheKey_requestEquality/same_tenant,_same_query_with_start_time_in_same_bucket === RUN Test_cardinalityEstimateBucket_GenerateCacheKey_requestEquality/same_tenant,_same_query_with_start_time_in_different_bucket === PAUSE Test_cardinalityEstimateBucket_GenerateCacheKey_requestEquality/same_tenant,_same_query_with_start_time_in_different_bucket === RUN Test_cardinalityEstimateBucket_GenerateCacheKey_requestEquality/same_tenant,_same_query_with_start_time_in_same_bucket_and_range_size_in_same_bucket === PAUSE Test_cardinalityEstimateBucket_GenerateCacheKey_requestEquality/same_tenant,_same_query_with_start_time_in_same_bucket_and_range_size_in_same_bucket === RUN Test_cardinalityEstimateBucket_GenerateCacheKey_requestEquality/same_tenant,_same_query_with_start_time_in_same_bucket_and_range_size_in_different_bucket === PAUSE Test_cardinalityEstimateBucket_GenerateCacheKey_requestEquality/same_tenant,_same_query_with_start_time_in_same_bucket_and_range_size_in_different_bucket === RUN Test_cardinalityEstimateBucket_GenerateCacheKey_requestEquality/same_tenant,_same_query_with_start_time_less_than_a_bucket_width_apart_but_in_different_buckets === PAUSE Test_cardinalityEstimateBucket_GenerateCacheKey_requestEquality/same_tenant,_same_query_with_start_time_less_than_a_bucket_width_apart_but_in_different_buckets === RUN Test_cardinalityEstimateBucket_GenerateCacheKey_requestEquality/same_tenant,_same_query_with_start_time_less_than_a_bucket_width_apart_and_in_the_same_bucket === PAUSE Test_cardinalityEstimateBucket_GenerateCacheKey_requestEquality/same_tenant,_same_query_with_start_time_less_than_a_bucket_width_apart_and_in_the_same_bucket === CONT Test_cardinalityEstimateBucket_GenerateCacheKey_requestEquality/same_tenant,_same_request === CONT Test_cardinalityEstimateBucket_GenerateCacheKey_requestEquality/same_tenant,_same_query_with_start_time_less_than_a_bucket_width_apart_and_in_the_same_bucket === CONT Test_cardinalityEstimateBucket_GenerateCacheKey_requestEquality/same_tenant,_same_query_with_start_time_in_same_bucket_and_range_size_in_same_bucket === CONT Test_cardinalityEstimateBucket_GenerateCacheKey_requestEquality/same_tenant,_same_query_with_start_time_less_than_a_bucket_width_apart_but_in_different_buckets === CONT Test_cardinalityEstimateBucket_GenerateCacheKey_requestEquality/same_tenant,_same_query_with_start_time_in_same_bucket === CONT Test_cardinalityEstimateBucket_GenerateCacheKey_requestEquality/same_tenant,_same_query_with_start_time_in_different_bucket === CONT Test_cardinalityEstimateBucket_GenerateCacheKey_requestEquality/different_tenant,_same_request === CONT Test_cardinalityEstimateBucket_GenerateCacheKey_requestEquality/same_tenant,_same_query_with_start_time_in_same_bucket_and_range_size_in_different_bucket --- PASS: Test_cardinalityEstimateBucket_GenerateCacheKey_requestEquality (0.01s) --- PASS: Test_cardinalityEstimateBucket_GenerateCacheKey_requestEquality/same_tenant,_same_request (0.00s) --- PASS: Test_cardinalityEstimateBucket_GenerateCacheKey_requestEquality/same_tenant,_same_query_with_start_time_less_than_a_bucket_width_apart_and_in_the_same_bucket (0.00s) --- PASS: Test_cardinalityEstimateBucket_GenerateCacheKey_requestEquality/same_tenant,_same_query_with_start_time_less_than_a_bucket_width_apart_but_in_different_buckets (0.00s) --- PASS: Test_cardinalityEstimateBucket_GenerateCacheKey_requestEquality/same_tenant,_same_query_with_start_time_in_same_bucket_and_range_size_in_same_bucket (0.00s) --- PASS: Test_cardinalityEstimateBucket_GenerateCacheKey_requestEquality/different_tenant,_same_request (0.00s) --- PASS: Test_cardinalityEstimateBucket_GenerateCacheKey_requestEquality/same_tenant,_same_query_with_start_time_in_same_bucket_and_range_size_in_different_bucket (0.00s) --- PASS: Test_cardinalityEstimateBucket_GenerateCacheKey_requestEquality/same_tenant,_same_query_with_start_time_in_same_bucket (0.00s) --- PASS: Test_cardinalityEstimateBucket_GenerateCacheKey_requestEquality/same_tenant,_same_query_with_start_time_in_different_bucket (0.00s) === RUN Test_newCardinalityEstimationMiddleware_canWrapMoreThanOnce --- PASS: Test_newCardinalityEstimationMiddleware_canWrapMoreThanOnce (0.00s) === RUN TestPrometheusCodec_JSONResponse === RUN TestPrometheusCodec_JSONResponse/successful_string_response === RUN TestPrometheusCodec_JSONResponse/successful_scalar_response === RUN TestPrometheusCodec_JSONResponse/successful_vector_response === RUN TestPrometheusCodec_JSONResponse/successful_matrix_response_with_float_values === RUN TestPrometheusCodec_JSONResponse/successful_empty_matrix_response === RUN TestPrometheusCodec_JSONResponse/error_response --- PASS: TestPrometheusCodec_JSONResponse (0.04s) --- PASS: TestPrometheusCodec_JSONResponse/successful_string_response (0.01s) --- PASS: TestPrometheusCodec_JSONResponse/successful_scalar_response (0.00s) --- PASS: TestPrometheusCodec_JSONResponse/successful_vector_response (0.02s) --- PASS: TestPrometheusCodec_JSONResponse/successful_matrix_response_with_float_values (0.01s) --- PASS: TestPrometheusCodec_JSONResponse/successful_empty_matrix_response (0.00s) --- PASS: TestPrometheusCodec_JSONResponse/error_response (0.00s) === RUN TestPrometheusCodec_JSONEncoding === RUN TestPrometheusCodec_JSONEncoding/successful_matrix_response_with_histogram_values === RUN TestPrometheusCodec_JSONEncoding/successful_matrix_response_with_a_single_series_with_both_float_and_histogram_values --- PASS: TestPrometheusCodec_JSONEncoding (0.01s) --- PASS: TestPrometheusCodec_JSONEncoding/successful_matrix_response_with_histogram_values (0.00s) --- PASS: TestPrometheusCodec_JSONEncoding/successful_matrix_response_with_a_single_series_with_both_float_and_histogram_values (0.00s) === RUN TestProtobufFormat_DecodeResponse === RUN TestProtobufFormat_DecodeResponse/successful_string_response === RUN TestProtobufFormat_DecodeResponse/successful_scalar_response === RUN TestProtobufFormat_DecodeResponse/successful_empty_vector_response === RUN TestProtobufFormat_DecodeResponse/successful_vector_response_with_single_series_with_no_labels === RUN TestProtobufFormat_DecodeResponse/successful_vector_response_with_single_series_with_one_label === RUN TestProtobufFormat_DecodeResponse/successful_vector_response_with_single_series_with_many_labels === RUN TestProtobufFormat_DecodeResponse/successful_vector_response_with_multiple_series === RUN TestProtobufFormat_DecodeResponse/successful_vector_response_with_histogram_value === RUN TestProtobufFormat_DecodeResponse/successful_vector_response_with_float_and_histogram_values === RUN TestProtobufFormat_DecodeResponse/successful_vector_response_with_malformed_metric_symbols === RUN TestProtobufFormat_DecodeResponse/successful_matrix_response_with_no_series === RUN TestProtobufFormat_DecodeResponse/successful_matrix_response_with_single_series_with_no_labels_and_no_samples === RUN TestProtobufFormat_DecodeResponse/successful_matrix_response_with_single_series_with_one_label_and_no_samples === RUN TestProtobufFormat_DecodeResponse/successful_matrix_response_with_single_series_with_many_labels_and_no_samples === RUN TestProtobufFormat_DecodeResponse/successful_matrix_response_with_single_series_with_one_sample === RUN TestProtobufFormat_DecodeResponse/successful_matrix_response_with_single_series_with_many_samples === RUN TestProtobufFormat_DecodeResponse/successful_matrix_response_with_multiple_series === RUN TestProtobufFormat_DecodeResponse/successful_matrix_response_with_histogram_value === RUN TestProtobufFormat_DecodeResponse/successful_matrix_response_with_float_and_histogram_values === RUN TestProtobufFormat_DecodeResponse/successful_matrix_response_with_malformed_metric_symbols === RUN TestProtobufFormat_DecodeResponse/successful_empty_matrix_response === RUN TestProtobufFormat_DecodeResponse/error_response --- PASS: TestProtobufFormat_DecodeResponse (0.03s) --- PASS: TestProtobufFormat_DecodeResponse/successful_string_response (0.00s) --- PASS: TestProtobufFormat_DecodeResponse/successful_scalar_response (0.00s) --- PASS: TestProtobufFormat_DecodeResponse/successful_empty_vector_response (0.00s) --- PASS: TestProtobufFormat_DecodeResponse/successful_vector_response_with_single_series_with_no_labels (0.00s) --- PASS: TestProtobufFormat_DecodeResponse/successful_vector_response_with_single_series_with_one_label (0.00s) --- PASS: TestProtobufFormat_DecodeResponse/successful_vector_response_with_single_series_with_many_labels (0.00s) --- PASS: TestProtobufFormat_DecodeResponse/successful_vector_response_with_multiple_series (0.00s) --- PASS: TestProtobufFormat_DecodeResponse/successful_vector_response_with_histogram_value (0.00s) --- PASS: TestProtobufFormat_DecodeResponse/successful_vector_response_with_float_and_histogram_values (0.00s) --- PASS: TestProtobufFormat_DecodeResponse/successful_vector_response_with_malformed_metric_symbols (0.00s) --- PASS: TestProtobufFormat_DecodeResponse/successful_matrix_response_with_no_series (0.00s) --- PASS: TestProtobufFormat_DecodeResponse/successful_matrix_response_with_single_series_with_no_labels_and_no_samples (0.00s) --- PASS: TestProtobufFormat_DecodeResponse/successful_matrix_response_with_single_series_with_one_label_and_no_samples (0.00s) --- PASS: TestProtobufFormat_DecodeResponse/successful_matrix_response_with_single_series_with_many_labels_and_no_samples (0.00s) --- PASS: TestProtobufFormat_DecodeResponse/successful_matrix_response_with_single_series_with_one_sample (0.00s) --- PASS: TestProtobufFormat_DecodeResponse/successful_matrix_response_with_single_series_with_many_samples (0.00s) --- PASS: TestProtobufFormat_DecodeResponse/successful_matrix_response_with_multiple_series (0.00s) --- PASS: TestProtobufFormat_DecodeResponse/successful_matrix_response_with_histogram_value (0.00s) --- PASS: TestProtobufFormat_DecodeResponse/successful_matrix_response_with_float_and_histogram_values (0.00s) --- PASS: TestProtobufFormat_DecodeResponse/successful_matrix_response_with_malformed_metric_symbols (0.00s) --- PASS: TestProtobufFormat_DecodeResponse/successful_empty_matrix_response (0.00s) --- PASS: TestProtobufFormat_DecodeResponse/error_response (0.00s) === RUN TestProtobufFormat_EncodeResponse === RUN TestProtobufFormat_EncodeResponse/successful_string_response === RUN TestProtobufFormat_EncodeResponse/successful_scalar_response === RUN TestProtobufFormat_EncodeResponse/successful_empty_vector_response === RUN TestProtobufFormat_EncodeResponse/successful_vector_response_with_single_series_with_no_labels === RUN TestProtobufFormat_EncodeResponse/successful_vector_response_with_single_series_with_one_label === RUN TestProtobufFormat_EncodeResponse/successful_vector_response_with_single_series_with_many_labels === RUN TestProtobufFormat_EncodeResponse/successful_vector_response_with_multiple_series === RUN TestProtobufFormat_EncodeResponse/successful_vector_response_with_histogram_value === RUN TestProtobufFormat_EncodeResponse/successful_vector_response_with_float_and_histogram_values === RUN TestProtobufFormat_EncodeResponse/successful_matrix_response_with_no_series === RUN TestProtobufFormat_EncodeResponse/successful_matrix_response_with_single_series_with_no_labels_and_no_samples === RUN TestProtobufFormat_EncodeResponse/successful_matrix_response_with_single_series_with_one_label_and_no_samples === RUN TestProtobufFormat_EncodeResponse/successful_matrix_response_with_single_series_with_many_labels_and_no_samples === RUN TestProtobufFormat_EncodeResponse/successful_matrix_response_with_single_series_with_one_sample === RUN TestProtobufFormat_EncodeResponse/successful_matrix_response_with_single_series_with_many_samples === RUN TestProtobufFormat_EncodeResponse/successful_matrix_response_with_multiple_series === RUN TestProtobufFormat_EncodeResponse/successful_matrix_response_with_histogram_value === RUN TestProtobufFormat_EncodeResponse/successful_matrix_response_with_float_and_histogram_values === RUN TestProtobufFormat_EncodeResponse/successful_empty_matrix_response === RUN TestProtobufFormat_EncodeResponse/error_response --- PASS: TestProtobufFormat_EncodeResponse (0.07s) --- PASS: TestProtobufFormat_EncodeResponse/successful_string_response (0.00s) --- PASS: TestProtobufFormat_EncodeResponse/successful_scalar_response (0.00s) --- PASS: TestProtobufFormat_EncodeResponse/successful_empty_vector_response (0.00s) --- PASS: TestProtobufFormat_EncodeResponse/successful_vector_response_with_single_series_with_no_labels (0.00s) --- PASS: TestProtobufFormat_EncodeResponse/successful_vector_response_with_single_series_with_one_label (0.00s) --- PASS: TestProtobufFormat_EncodeResponse/successful_vector_response_with_single_series_with_many_labels (0.00s) --- PASS: TestProtobufFormat_EncodeResponse/successful_vector_response_with_multiple_series (0.00s) --- PASS: TestProtobufFormat_EncodeResponse/successful_vector_response_with_histogram_value (0.00s) --- PASS: TestProtobufFormat_EncodeResponse/successful_vector_response_with_float_and_histogram_values (0.00s) --- PASS: TestProtobufFormat_EncodeResponse/successful_matrix_response_with_no_series (0.00s) --- PASS: TestProtobufFormat_EncodeResponse/successful_matrix_response_with_single_series_with_no_labels_and_no_samples (0.00s) --- PASS: TestProtobufFormat_EncodeResponse/successful_matrix_response_with_single_series_with_one_label_and_no_samples (0.00s) --- PASS: TestProtobufFormat_EncodeResponse/successful_matrix_response_with_single_series_with_many_labels_and_no_samples (0.00s) --- PASS: TestProtobufFormat_EncodeResponse/successful_matrix_response_with_single_series_with_one_sample (0.00s) --- PASS: TestProtobufFormat_EncodeResponse/successful_matrix_response_with_single_series_with_many_samples (0.00s) --- PASS: TestProtobufFormat_EncodeResponse/successful_matrix_response_with_multiple_series (0.00s) --- PASS: TestProtobufFormat_EncodeResponse/successful_matrix_response_with_histogram_value (0.01s) --- PASS: TestProtobufFormat_EncodeResponse/successful_matrix_response_with_float_and_histogram_values (0.00s) --- PASS: TestProtobufFormat_EncodeResponse/successful_empty_matrix_response (0.00s) --- PASS: TestProtobufFormat_EncodeResponse/error_response (0.01s) === RUN TestRequest === RUN TestRequest/0 === RUN TestRequest/1 === RUN TestRequest/2 === RUN TestRequest/3 === RUN TestRequest/4 === RUN TestRequest/5 === RUN TestRequest/6 === RUN TestRequest/7 --- PASS: TestRequest (0.04s) --- PASS: TestRequest/0 (0.01s) --- PASS: TestRequest/1 (0.00s) --- PASS: TestRequest/2 (0.00s) --- PASS: TestRequest/3 (0.00s) --- PASS: TestRequest/4 (0.00s) --- PASS: TestRequest/5 (0.00s) --- PASS: TestRequest/6 (0.00s) --- PASS: TestRequest/7 (0.00s) === RUN TestPrometheusCodec_EncodeRequest_AcceptHeader === RUN TestPrometheusCodec_EncodeRequest_AcceptHeader/json === RUN TestPrometheusCodec_EncodeRequest_AcceptHeader/protobuf --- PASS: TestPrometheusCodec_EncodeRequest_AcceptHeader (0.00s) --- PASS: TestPrometheusCodec_EncodeRequest_AcceptHeader/json (0.00s) --- PASS: TestPrometheusCodec_EncodeRequest_AcceptHeader/protobuf (0.00s) === RUN TestPrometheusCodec_EncodeResponse_ContentNegotiation === RUN TestPrometheusCodec_EncodeResponse_ContentNegotiation/multiple_unsupported_content_types_in_Accept_header === RUN TestPrometheusCodec_EncodeResponse_ContentNegotiation/single_supported_content_type_in_Accept_header === RUN TestPrometheusCodec_EncodeResponse_ContentNegotiation/wildcard_subtype_in_Accept_header === RUN TestPrometheusCodec_EncodeResponse_ContentNegotiation/wildcard_in_Accept_header === RUN TestPrometheusCodec_EncodeResponse_ContentNegotiation/multiple_supported_content_types_in_Accept_header === RUN TestPrometheusCodec_EncodeResponse_ContentNegotiation/no_content_type_in_Accept_header === RUN TestPrometheusCodec_EncodeResponse_ContentNegotiation/unsupported_content_type_in_Accept_header --- PASS: TestPrometheusCodec_EncodeResponse_ContentNegotiation (0.00s) --- PASS: TestPrometheusCodec_EncodeResponse_ContentNegotiation/multiple_unsupported_content_types_in_Accept_header (0.00s) --- PASS: TestPrometheusCodec_EncodeResponse_ContentNegotiation/single_supported_content_type_in_Accept_header (0.00s) --- PASS: TestPrometheusCodec_EncodeResponse_ContentNegotiation/wildcard_subtype_in_Accept_header (0.00s) --- PASS: TestPrometheusCodec_EncodeResponse_ContentNegotiation/wildcard_in_Accept_header (0.00s) --- PASS: TestPrometheusCodec_EncodeResponse_ContentNegotiation/multiple_supported_content_types_in_Accept_header (0.00s) --- PASS: TestPrometheusCodec_EncodeResponse_ContentNegotiation/no_content_type_in_Accept_header (0.00s) --- PASS: TestPrometheusCodec_EncodeResponse_ContentNegotiation/unsupported_content_type_in_Accept_header (0.00s) === RUN TestDecodeFailedResponse === RUN TestDecodeFailedResponse/internal_error === RUN TestDecodeFailedResponse/too_many_requests === RUN TestDecodeFailedResponse/too_large_entry --- PASS: TestDecodeFailedResponse (0.00s) --- PASS: TestDecodeFailedResponse/internal_error (0.00s) --- PASS: TestDecodeFailedResponse/too_many_requests (0.00s) --- PASS: TestDecodeFailedResponse/too_large_entry (0.00s) === RUN TestPrometheusCodec_DecodeResponse_ContentTypeHandling === RUN TestPrometheusCodec_DecodeResponse_ContentTypeHandling/unknown_content_type_in_response === RUN TestPrometheusCodec_DecodeResponse_ContentTypeHandling/no_content_type_in_response --- PASS: TestPrometheusCodec_DecodeResponse_ContentTypeHandling (0.00s) --- PASS: TestPrometheusCodec_DecodeResponse_ContentTypeHandling/unknown_content_type_in_response (0.00s) --- PASS: TestPrometheusCodec_DecodeResponse_ContentTypeHandling/no_content_type_in_response (0.00s) === RUN TestMergeAPIResponses === RUN TestMergeAPIResponses/No_responses_shouldn't_panic_and_return_a_non-null_result_and_result_type. === RUN TestMergeAPIResponses/A_single_empty_response_shouldn't_panic. === RUN TestMergeAPIResponses/Multiple_empty_responses_shouldn't_panic. === RUN TestMergeAPIResponses/Basic_merging_of_two_responses. === RUN TestMergeAPIResponses/Merging_of_responses_when_labels_are_in_different_order. === RUN TestMergeAPIResponses/Merging_of_samples_where_there_is_single_overlap. === RUN TestMergeAPIResponses/Merging_of_samples_where_there_is_multiple_partial_overlaps. === RUN TestMergeAPIResponses/Merging_of_samples_where_there_is_complete_overlap. === RUN TestMergeAPIResponses/Handling_single_histogram_result === RUN TestMergeAPIResponses/Handling_non_overlapping_histogram_result === RUN TestMergeAPIResponses/shouldn't_merge_unsuccessful_responses === RUN TestMergeAPIResponses/shouldn't_merge_nil_data === RUN TestMergeAPIResponses/shouldn't_merge_non-matrix_data --- PASS: TestMergeAPIResponses (0.02s) --- PASS: TestMergeAPIResponses/No_responses_shouldn't_panic_and_return_a_non-null_result_and_result_type. (0.00s) --- PASS: TestMergeAPIResponses/A_single_empty_response_shouldn't_panic. (0.00s) --- PASS: TestMergeAPIResponses/Multiple_empty_responses_shouldn't_panic. (0.00s) --- PASS: TestMergeAPIResponses/Basic_merging_of_two_responses. (0.00s) --- PASS: TestMergeAPIResponses/Merging_of_responses_when_labels_are_in_different_order. (0.00s) --- PASS: TestMergeAPIResponses/Merging_of_samples_where_there_is_single_overlap. (0.00s) --- PASS: TestMergeAPIResponses/Merging_of_samples_where_there_is_multiple_partial_overlaps. (0.00s) --- PASS: TestMergeAPIResponses/Merging_of_samples_where_there_is_complete_overlap. (0.00s) --- PASS: TestMergeAPIResponses/Handling_single_histogram_result (0.01s) --- PASS: TestMergeAPIResponses/Handling_non_overlapping_histogram_result (0.00s) --- PASS: TestMergeAPIResponses/shouldn't_merge_unsuccessful_responses (0.00s) --- PASS: TestMergeAPIResponses/shouldn't_merge_nil_data (0.00s) --- PASS: TestMergeAPIResponses/shouldn't_merge_non-matrix_data (0.00s) === RUN Test_DecodeOptions === RUN Test_DecodeOptions/default === PAUSE Test_DecodeOptions/default === RUN Test_DecodeOptions/disable_cache === PAUSE Test_DecodeOptions/disable_cache === RUN Test_DecodeOptions/custom_sharding === PAUSE Test_DecodeOptions/custom_sharding === RUN Test_DecodeOptions/disable_sharding === PAUSE Test_DecodeOptions/disable_sharding === RUN Test_DecodeOptions/custom_instant_query_splitting === PAUSE Test_DecodeOptions/custom_instant_query_splitting === RUN Test_DecodeOptions/disable_instant_query_splitting === PAUSE Test_DecodeOptions/disable_instant_query_splitting === CONT Test_DecodeOptions/default === CONT Test_DecodeOptions/disable_sharding === CONT Test_DecodeOptions/custom_sharding === CONT Test_DecodeOptions/disable_instant_query_splitting === CONT Test_DecodeOptions/custom_instant_query_splitting === CONT Test_DecodeOptions/disable_cache --- PASS: Test_DecodeOptions (0.00s) --- PASS: Test_DecodeOptions/default (0.00s) --- PASS: Test_DecodeOptions/disable_sharding (0.00s) --- PASS: Test_DecodeOptions/custom_sharding (0.00s) --- PASS: Test_DecodeOptions/disable_instant_query_splitting (0.00s) --- PASS: Test_DecodeOptions/custom_instant_query_splitting (0.00s) --- PASS: Test_DecodeOptions/disable_cache (0.00s) === RUN TestLimitsMiddleware_MaxQueryLookback === RUN TestLimitsMiddleware_MaxQueryLookback/should_not_manipulate_time_range_if_max_lookback_is_disabled === RUN TestLimitsMiddleware_MaxQueryLookback/should_not_manipulate_time_range_for_a_query_on_short_time_range === RUN TestLimitsMiddleware_MaxQueryLookback/should_not_manipulate_a_query_on_large_time_range_close_to_the_limit === RUN TestLimitsMiddleware_MaxQueryLookback/should_manipulate_a_query_on_large_time_range_over_the_limit === RUN TestLimitsMiddleware_MaxQueryLookback/should_skip_executing_a_query_outside_the_allowed_time_range === RUN TestLimitsMiddleware_MaxQueryLookback/should_manipulate_a_query_where_maxQueryLookback_is_past_the_retention_period --- PASS: TestLimitsMiddleware_MaxQueryLookback (0.01s) --- PASS: TestLimitsMiddleware_MaxQueryLookback/should_not_manipulate_time_range_if_max_lookback_is_disabled (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLookback/should_not_manipulate_time_range_for_a_query_on_short_time_range (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLookback/should_not_manipulate_a_query_on_large_time_range_close_to_the_limit (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLookback/should_manipulate_a_query_on_large_time_range_over_the_limit (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLookback/should_skip_executing_a_query_outside_the_allowed_time_range (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLookback/should_manipulate_a_query_where_maxQueryLookback_is_past_the_retention_period (0.00s) === RUN TestLimitsMiddleware_MaxQueryExpressionSizeBytes === RUN TestLimitsMiddleware_MaxQueryExpressionSizeBytes/should_work_for_queries_under_the_limit === RUN TestLimitsMiddleware_MaxQueryExpressionSizeBytes/should_work_for_queries_when_the_limit_is_disabled === RUN TestLimitsMiddleware_MaxQueryExpressionSizeBytes/should_fail_for_queries_longer_than_the_limit === RUN TestLimitsMiddleware_MaxQueryExpressionSizeBytes/should_fail_for_queries_longer_than_a_one_tenant_limit === RUN TestLimitsMiddleware_MaxQueryExpressionSizeBytes/should_fail_for_queries_longer_than_a_one_tenant_limit_with_one_limit_disabled --- PASS: TestLimitsMiddleware_MaxQueryExpressionSizeBytes (0.01s) --- PASS: TestLimitsMiddleware_MaxQueryExpressionSizeBytes/should_work_for_queries_under_the_limit (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryExpressionSizeBytes/should_work_for_queries_when_the_limit_is_disabled (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryExpressionSizeBytes/should_fail_for_queries_longer_than_the_limit (0.01s) --- PASS: TestLimitsMiddleware_MaxQueryExpressionSizeBytes/should_fail_for_queries_longer_than_a_one_tenant_limit (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryExpressionSizeBytes/should_fail_for_queries_longer_than_a_one_tenant_limit_with_one_limit_disabled (0.00s) === RUN TestLimitsMiddleware_MaxQueryLength === RUN TestLimitsMiddleware_MaxQueryLength/should_skip_validation_if_max_length_is_disabled === RUN TestLimitsMiddleware_MaxQueryLength/should_succeed_on_a_query_on_short_time_range,_ending_now === RUN TestLimitsMiddleware_MaxQueryLength/should_succeed_on_a_query_on_short_time_range,_ending_in_the_past === RUN TestLimitsMiddleware_MaxQueryLength/should_succeed_on_a_query_on_large_time_range_close_to_the_limit,_ending_now === RUN TestLimitsMiddleware_MaxQueryLength/should_fail_on_a_query_on_large_time_range_over_the_limit,_ending_now === RUN TestLimitsMiddleware_MaxQueryLength/should_fail_on_a_query_on_large_time_range_over_the_limit,_ending_in_the_past === RUN TestLimitsMiddleware_MaxQueryLength/should_succeed_if_total_query_length_is_higher_than_query_length_limit --- PASS: TestLimitsMiddleware_MaxQueryLength (0.01s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_skip_validation_if_max_length_is_disabled (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_succeed_on_a_query_on_short_time_range,_ending_now (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_succeed_on_a_query_on_short_time_range,_ending_in_the_past (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_succeed_on_a_query_on_large_time_range_close_to_the_limit,_ending_now (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_fail_on_a_query_on_large_time_range_over_the_limit,_ending_now (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_fail_on_a_query_on_large_time_range_over_the_limit,_ending_in_the_past (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_succeed_if_total_query_length_is_higher_than_query_length_limit (0.00s) === RUN TestLimitsMiddleware_CreationGracePeriod === RUN TestLimitsMiddleware_CreationGracePeriod/should_not_manipulate_time_range_if_creation_grace_period_is_disabled === RUN TestLimitsMiddleware_CreationGracePeriod/should_not_manipulate_time_range_for_a_query_in_now_+_creation_grace_period === RUN TestLimitsMiddleware_CreationGracePeriod/should_manipulate_time_range_for_a_query_over_now_+_creation_grace_period --- PASS: TestLimitsMiddleware_CreationGracePeriod (0.02s) --- PASS: TestLimitsMiddleware_CreationGracePeriod/should_not_manipulate_time_range_if_creation_grace_period_is_disabled (0.01s) --- PASS: TestLimitsMiddleware_CreationGracePeriod/should_not_manipulate_time_range_for_a_query_in_now_+_creation_grace_period (0.00s) --- PASS: TestLimitsMiddleware_CreationGracePeriod/should_manipulate_time_range_for_a_query_over_now_+_creation_grace_period (0.00s) === RUN TestLimitedRoundTripper_MaxQueryParallelism --- PASS: TestLimitedRoundTripper_MaxQueryParallelism (0.24s) === RUN TestLimitedRoundTripper_MaxQueryParallelismLateScheduling --- PASS: TestLimitedRoundTripper_MaxQueryParallelismLateScheduling (0.02s) === RUN TestLimitedRoundTripper_OriginalRequestContextCancellation --- PASS: TestLimitedRoundTripper_OriginalRequestContextCancellation (0.10s) === RUN TestQueryShardingCorrectness === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_sum === PAUSE TestQueryShardingCorrectness/outer_subquery_on_top_of_sum === RUN TestQueryShardingCorrectness/bottomk() === PAUSE TestQueryShardingCorrectness/bottomk() === RUN TestQueryShardingCorrectness/histogram_count(sum(metric_native_histogram)) === PAUSE TestQueryShardingCorrectness/histogram_count(sum(metric_native_histogram)) === RUN TestQueryShardingCorrectness/sum()_grouping_'by' === PAUSE TestQueryShardingCorrectness/sum()_grouping_'by' === RUN TestQueryShardingCorrectness/sum()_grouping_'without' === PAUSE TestQueryShardingCorrectness/sum()_grouping_'without' === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by' === PAUSE TestQueryShardingCorrectness/histogram_quantile()_grouping_'by' === RUN TestQueryShardingCorrectness/max()_grouping_'by' === PAUSE TestQueryShardingCorrectness/max()_grouping_'by' === RUN TestQueryShardingCorrectness/and === PAUSE TestQueryShardingCorrectness/and === RUN TestQueryShardingCorrectness/histogram_quantile(0.5,_sum(metric_native_histogram)) === PAUSE TestQueryShardingCorrectness/histogram_quantile(0.5,_sum(metric_native_histogram)) === RUN TestQueryShardingCorrectness/count()_grouping_'by' === PAUSE TestQueryShardingCorrectness/count()_grouping_'by' === RUN TestQueryShardingCorrectness/count()_grouping_'without' === PAUSE TestQueryShardingCorrectness/count()_grouping_'without' === RUN TestQueryShardingCorrectness/subquery_min === PAUSE TestQueryShardingCorrectness/subquery_min === RUN TestQueryShardingCorrectness/count(metric_native_histogram) === PAUSE TestQueryShardingCorrectness/count(metric_native_histogram) === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1) === PAUSE TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1) === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1 === PAUSE TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1 === RUN TestQueryShardingCorrectness/sum({__name__!=""})_excluding_native_histograms === PAUSE TestQueryShardingCorrectness/sum({__name__!=""})_excluding_native_histograms === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})_excluding_native_histograms === PAUSE TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})_excluding_native_histograms === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))_excluding_native_histograms === PAUSE TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))_excluding_native_histograms === RUN TestQueryShardingCorrectness/sum(count()) === PAUSE TestQueryShardingCorrectness/sum(count()) === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar === PAUSE TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant === PAUSE TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant === RUN TestQueryShardingCorrectness/count_by_(group_1)_(metric_native_histogram) === PAUSE TestQueryShardingCorrectness/count_by_(group_1)_(metric_native_histogram) === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without' === PAUSE TestQueryShardingCorrectness/sum(rate())_grouping_'without' === RUN TestQueryShardingCorrectness/sum(avg_over_time()) === PAUSE TestQueryShardingCorrectness/sum(avg_over_time()) === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand === PAUSE TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping === PAUSE TestQueryShardingCorrectness/histogram_quantile()_no_grouping === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1) === PAUSE TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1) === RUN TestQueryShardingCorrectness/string_literal === PAUSE TestQueryShardingCorrectness/string_literal === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1) === PAUSE TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1) === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(metric_native_histogram) === PAUSE TestQueryShardingCorrectness/sum_by_(group_1)_(metric_native_histogram) === RUN TestQueryShardingCorrectness/count_by_(group_1)_(count_over_time(metric_native_histogram[1m])) === PAUSE TestQueryShardingCorrectness/count_by_(group_1)_(count_over_time(metric_native_histogram[1m])) === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children === PAUSE TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children === RUN TestQueryShardingCorrectness/month()_>=_1_and_month() === PAUSE TestQueryShardingCorrectness/month()_>=_1_and_month() === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping === PAUSE TestQueryShardingCorrectness/sum(rate())_no_grouping === RUN TestQueryShardingCorrectness/or === PAUSE TestQueryShardingCorrectness/or === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate()) === PAUSE TestQueryShardingCorrectness/sum(rate())_>_avg(rate()) === RUN TestQueryShardingCorrectness/sum_of_subquery_min === PAUSE TestQueryShardingCorrectness/sum_of_subquery_min === RUN TestQueryShardingCorrectness/@_modifier === PAUSE TestQueryShardingCorrectness/@_modifier === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order === PAUSE TestQueryShardingCorrectness/query_with_sort()_expects_specific_order === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time(metric_native_histogram[1m])) === PAUSE TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time(metric_native_histogram[1m])) === RUN TestQueryShardingCorrectness/sum()_offset === PAUSE TestQueryShardingCorrectness/sum()_offset === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by' === PAUSE TestQueryShardingCorrectness/sum(rate())_grouping_'by' === RUN TestQueryShardingCorrectness/count()_no_grouping === PAUSE TestQueryShardingCorrectness/count()_no_grouping === RUN TestQueryShardingCorrectness/sum(min_over_time()) === PAUSE TestQueryShardingCorrectness/sum(min_over_time()) === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month() === PAUSE TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month() === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr === PAUSE TestQueryShardingCorrectness/subquery_min_over_time_with_aggr === RUN TestQueryShardingCorrectness/histogram_fraction(0,_0.5,_sum(metric_native_histogram)) === PAUSE TestQueryShardingCorrectness/histogram_fraction(0,_0.5,_sum(metric_native_histogram)) === RUN TestQueryShardingCorrectness/sum()_no_grouping === PAUSE TestQueryShardingCorrectness/sum()_no_grouping === RUN TestQueryShardingCorrectness/max()_no_grouping === PAUSE TestQueryShardingCorrectness/max()_no_grouping === RUN TestQueryShardingCorrectness/avg()_grouping_'by' === PAUSE TestQueryShardingCorrectness/avg()_grouping_'by' === RUN TestQueryShardingCorrectness/nested_count() === PAUSE TestQueryShardingCorrectness/nested_count() === RUN TestQueryShardingCorrectness/scalar(aggregation) === PAUSE TestQueryShardingCorrectness/scalar(aggregation) === RUN TestQueryShardingCorrectness/label_replace === PAUSE TestQueryShardingCorrectness/label_replace === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_avg === PAUSE TestQueryShardingCorrectness/outer_subquery_on_top_of_avg === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1) === PAUSE TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1) === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset === PAUSE TestQueryShardingCorrectness/@_modifier_and_negative_offset === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation === PAUSE TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1 === PAUSE TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1 === RUN TestQueryShardingCorrectness/min()_no_grouping === PAUSE TestQueryShardingCorrectness/min()_no_grouping === RUN TestQueryShardingCorrectness/min()_grouping_'without' === PAUSE TestQueryShardingCorrectness/min()_grouping_'without' === RUN TestQueryShardingCorrectness/avg()_grouping_'without' === PAUSE TestQueryShardingCorrectness/avg()_grouping_'without' === RUN TestQueryShardingCorrectness/subquery_max === PAUSE TestQueryShardingCorrectness/subquery_max === RUN TestQueryShardingCorrectness/triple_subquery === PAUSE TestQueryShardingCorrectness/triple_subquery === RUN TestQueryShardingCorrectness/histogram_sum(sum(metric_native_histogram)) === PAUSE TestQueryShardingCorrectness/histogram_sum(sum(metric_native_histogram)) === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without' === PAUSE TestQueryShardingCorrectness/histogram_quantile()_grouping_'without' === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series === PAUSE TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series === RUN TestQueryShardingCorrectness/avg()_no_grouping === PAUSE TestQueryShardingCorrectness/avg()_no_grouping === RUN TestQueryShardingCorrectness/time() === PAUSE TestQueryShardingCorrectness/time() === RUN TestQueryShardingCorrectness/sum(metric_native_histogram) === PAUSE TestQueryShardingCorrectness/sum(metric_native_histogram) === RUN TestQueryShardingCorrectness/topk() === PAUSE TestQueryShardingCorrectness/topk() === RUN TestQueryShardingCorrectness/scalar(single_metric) === PAUSE TestQueryShardingCorrectness/scalar(single_metric) === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month() === PAUSE TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month() === RUN TestQueryShardingCorrectness/sum()_negative_offset === PAUSE TestQueryShardingCorrectness/sum()_negative_offset === RUN TestQueryShardingCorrectness/sum(max_over_time()) === PAUSE TestQueryShardingCorrectness/sum(max_over_time()) === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter)) === PAUSE TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter)) === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar === PAUSE TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand === PAUSE TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand === RUN TestQueryShardingCorrectness/month(sum(metric_counter)) === PAUSE TestQueryShardingCorrectness/month(sum(metric_counter)) === RUN TestQueryShardingCorrectness/min()_grouping_'by' === PAUSE TestQueryShardingCorrectness/min()_grouping_'by' === RUN TestQueryShardingCorrectness/max()_grouping_'without' === PAUSE TestQueryShardingCorrectness/max()_grouping_'without' === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2 === PAUSE TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2 === RUN TestQueryShardingCorrectness/stddev() === PAUSE TestQueryShardingCorrectness/stddev() === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series === PAUSE TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le === PAUSE TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le === RUN TestQueryShardingCorrectness/@_modifier_and_offset === PAUSE TestQueryShardingCorrectness/@_modifier_and_offset === RUN TestQueryShardingCorrectness/vector() === PAUSE TestQueryShardingCorrectness/vector() === RUN TestQueryShardingCorrectness/0_<_bool_1 === PAUSE TestQueryShardingCorrectness/0_<_bool_1 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation === PAUSE TestQueryShardingCorrectness/histogram_quantile_without_aggregation === RUN TestQueryShardingCorrectness/vector(1) === PAUSE TestQueryShardingCorrectness/vector(1) === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2 === PAUSE TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2 === RUN TestQueryShardingCorrectness/double_subquery_deriv === PAUSE TestQueryShardingCorrectness/double_subquery_deriv === RUN TestQueryShardingCorrectness/label_join === PAUSE TestQueryShardingCorrectness/label_join === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand === PAUSE TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand === RUN TestQueryShardingCorrectness/stdvar() === PAUSE TestQueryShardingCorrectness/stdvar() === CONT TestQueryShardingCorrectness/outer_subquery_on_top_of_sum === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest === CONT TestQueryShardingCorrectness/sum()_no_grouping === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/stdvar() === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/@_modifier_and_offset === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/label_join === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/double_subquery_deriv === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/vector(1) === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/histogram_quantile_without_aggregation === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/0_<_bool_1 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/vector() === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/stddev() === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/max()_grouping_'without' === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/min()_grouping_'by' === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/month(sum(metric_counter)) === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/topk() === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter)) === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum(max_over_time()) === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum()_negative_offset === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month() === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/scalar(single_metric) === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum(metric_native_histogram) === RUN TestQueryShardingCorrectness/sum(metric_native_histogram)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(metric_native_histogram)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(metric_native_histogram)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(metric_native_histogram)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(metric_native_histogram)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(metric_native_histogram)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(metric_native_histogram)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(metric_native_histogram)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(metric_native_histogram)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(metric_native_histogram)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/time() === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/avg()_no_grouping === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/histogram_quantile()_grouping_'without' === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === CONT TestQueryShardingCorrectness/histogram_sum(sum(metric_native_histogram)) === RUN TestQueryShardingCorrectness/histogram_sum(sum(metric_native_histogram))/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/histogram_sum(sum(metric_native_histogram))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_sum(sum(metric_native_histogram))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_sum(sum(metric_native_histogram))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_sum(sum(metric_native_histogram))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_sum(sum(metric_native_histogram))/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/histogram_sum(sum(metric_native_histogram))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_sum(sum(metric_native_histogram))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_sum(sum(metric_native_histogram))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_sum(sum(metric_native_histogram))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/triple_subquery === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest === CONT TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1) === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/subquery_max === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/avg()_grouping_'without' === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/min()_grouping_'without' === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/min()_no_grouping === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest === CONT TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/@_modifier_and_negative_offset === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/outer_subquery_on_top_of_avg === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/label_replace === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/scalar(aggregation) === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/nested_count() === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/avg()_grouping_'by' === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/max()_no_grouping === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/histogram_quantile()_no_grouping === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/histogram_fraction(0,_0.5,_sum(metric_native_histogram)) === RUN TestQueryShardingCorrectness/histogram_fraction(0,_0.5,_sum(metric_native_histogram))/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/histogram_fraction(0,_0.5,_sum(metric_native_histogram))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_fraction(0,_0.5,_sum(metric_native_histogram))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_fraction(0,_0.5,_sum(metric_native_histogram))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_fraction(0,_0.5,_sum(metric_native_histogram))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_fraction(0,_0.5,_sum(metric_native_histogram))/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_fraction(0,_0.5,_sum(metric_native_histogram))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_fraction(0,_0.5,_sum(metric_native_histogram))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_fraction(0,_0.5,_sum(metric_native_histogram))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_fraction(0,_0.5,_sum(metric_native_histogram))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/query_with_sort()_expects_specific_order === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/subquery_min_over_time_with_aggr === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month() === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum(min_over_time()) === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/count()_no_grouping === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum(rate())_grouping_'by' === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum()_offset === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time(metric_native_histogram[1m])) === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time(metric_native_histogram[1m]))/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time(metric_native_histogram[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time(metric_native_histogram[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time(metric_native_histogram[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time(metric_native_histogram[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time(metric_native_histogram[1m]))/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time(metric_native_histogram[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time(metric_native_histogram[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time(metric_native_histogram[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time(metric_native_histogram[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/@_modifier === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum_of_subquery_min === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum(rate())_>_avg(rate()) === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest === CONT TestQueryShardingCorrectness/or === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/bottomk() === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/sum(rate())_no_grouping === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/month()_>=_1_and_month() === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/count_by_(group_1)_(count_over_time(metric_native_histogram[1m])) === RUN TestQueryShardingCorrectness/count_by_(group_1)_(count_over_time(metric_native_histogram[1m]))/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/count_by_(group_1)_(count_over_time(metric_native_histogram[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count_by_(group_1)_(count_over_time(metric_native_histogram[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/count_by_(group_1)_(count_over_time(metric_native_histogram[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/count_by_(group_1)_(count_over_time(metric_native_histogram[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/count_by_(group_1)_(count_over_time(metric_native_histogram[1m]))/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/count_by_(group_1)_(count_over_time(metric_native_histogram[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count_by_(group_1)_(count_over_time(metric_native_histogram[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/count_by_(group_1)_(count_over_time(metric_native_histogram[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/count_by_(group_1)_(count_over_time(metric_native_histogram[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum_by_(group_1)_(metric_native_histogram) === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(metric_native_histogram)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(metric_native_histogram)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1) === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(metric_native_histogram)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(metric_native_histogram)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(metric_native_histogram)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(metric_native_histogram)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/string_literal === RUN TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1) === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(metric_native_histogram)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(metric_native_histogram)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(metric_native_histogram)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(metric_native_histogram)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1) === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum(avg_over_time()) === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum(rate())_grouping_'without' === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/count_by_(group_1)_(metric_native_histogram) === RUN TestQueryShardingCorrectness/count_by_(group_1)_(metric_native_histogram)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/count_by_(group_1)_(metric_native_histogram)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count_by_(group_1)_(metric_native_histogram)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/count_by_(group_1)_(metric_native_histogram)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/count_by_(group_1)_(metric_native_histogram)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/count_by_(group_1)_(metric_native_histogram)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/count_by_(group_1)_(metric_native_histogram)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count_by_(group_1)_(metric_native_histogram)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/count_by_(group_1)_(metric_native_histogram)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/count_by_(group_1)_(metric_native_histogram)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum(count()) === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))_excluding_native_histograms === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))_excluding_native_histograms/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))_excluding_native_histograms/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))_excluding_native_histograms/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))_excluding_native_histograms/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))_excluding_native_histograms/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))_excluding_native_histograms/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))_excluding_native_histograms/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))_excluding_native_histograms/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))_excluding_native_histograms/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))_excluding_native_histograms/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})_excluding_native_histograms === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})_excluding_native_histograms/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})_excluding_native_histograms/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})_excluding_native_histograms/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})_excluding_native_histograms/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})_excluding_native_histograms/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})_excluding_native_histograms/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})_excluding_native_histograms/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})_excluding_native_histograms/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})_excluding_native_histograms/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})_excluding_native_histograms/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum({__name__!=""})_excluding_native_histograms === RUN TestQueryShardingCorrectness/sum({__name__!=""})_excluding_native_histograms/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum({__name__!=""})_excluding_native_histograms/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum({__name__!=""})_excluding_native_histograms/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum({__name__!=""})_excluding_native_histograms/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum({__name__!=""})_excluding_native_histograms/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum({__name__!=""})_excluding_native_histograms/*querymiddleware.PrometheusRangeQueryRequest === CONT TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum({__name__!=""})_excluding_native_histograms/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum({__name__!=""})_excluding_native_histograms/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/and === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum({__name__!=""})_excluding_native_histograms/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum({__name__!=""})_excluding_native_histograms/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/count(metric_native_histogram) === RUN TestQueryShardingCorrectness/count(metric_native_histogram)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/count(metric_native_histogram)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count(metric_native_histogram)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/count(metric_native_histogram)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/count(metric_native_histogram)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/count(metric_native_histogram)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/count(metric_native_histogram)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count(metric_native_histogram)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/count(metric_native_histogram)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/count(metric_native_histogram)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/subquery_min === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest === CONT TestQueryShardingCorrectness/count()_grouping_'without' === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest === CONT TestQueryShardingCorrectness/count()_grouping_'by' === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/histogram_quantile(0.5,_sum(metric_native_histogram)) === RUN TestQueryShardingCorrectness/histogram_quantile(0.5,_sum(metric_native_histogram))/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile(0.5,_sum(metric_native_histogram))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile(0.5,_sum(metric_native_histogram))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile(0.5,_sum(metric_native_histogram))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile(0.5,_sum(metric_native_histogram))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile(0.5,_sum(metric_native_histogram))/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile(0.5,_sum(metric_native_histogram))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile(0.5,_sum(metric_native_histogram))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile(0.5,_sum(metric_native_histogram))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile(0.5,_sum(metric_native_histogram))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/sum()_grouping_'without' === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/max()_grouping_'by' === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/histogram_quantile()_grouping_'by' === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum()_grouping_'by' === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/histogram_count(sum(metric_native_histogram)) === RUN TestQueryShardingCorrectness/histogram_count(sum(metric_native_histogram))/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/histogram_count(sum(metric_native_histogram))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_count(sum(metric_native_histogram))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_count(sum(metric_native_histogram))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_count(sum(metric_native_histogram))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_count(sum(metric_native_histogram))/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/histogram_count(sum(metric_native_histogram))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_count(sum(metric_native_histogram))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_count(sum(metric_native_histogram))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_count(sum(metric_native_histogram))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 --- PASS: TestQueryShardingCorrectness (2.36s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_sum (0.49s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest (0.49s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.09s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.09s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.05s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.18s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping (2.26s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest (0.50s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.05s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.07s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.09s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.26s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest (1.76s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.39s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.37s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.35s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.26s) --- PASS: TestQueryShardingCorrectness/stdvar() (2.47s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest (0.17s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.03s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.03s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.06s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest (2.30s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.92s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.24s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.18s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.36s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset (2.46s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest (0.90s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.16s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.16s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.17s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.30s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest (1.56s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.34s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.18s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.26s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.70s) --- PASS: TestQueryShardingCorrectness/label_join (1.97s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest (0.82s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.08s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.10s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.21s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.30s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest (1.15s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.16s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.21s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.23s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.32s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv (33.13s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest (3.94s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.56s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.55s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.99s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (1.63s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest (29.19s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (7.30s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (8.17s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (6.65s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (6.42s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2 (4.89s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest (1.14s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.08s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.19s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.14s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.66s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest (3.74s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.52s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.41s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.52s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.56s) --- PASS: TestQueryShardingCorrectness/vector(1) (0.04s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest (0.02s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest (0.03s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation (6.36s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest (0.43s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.06s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.06s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.10s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.15s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest (5.92s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (1.81s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (1.30s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.91s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (1.26s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1 (0.06s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest (0.04s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest (0.02s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand (49.63s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest (2.28s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.34s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.36s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.87s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.62s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest (47.34s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (11.01s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (11.21s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (9.84s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (10.84s) --- PASS: TestQueryShardingCorrectness/vector() (0.06s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest (0.01s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest (0.04s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series (6.84s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest (0.48s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.04s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.06s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.16s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.20s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest (6.35s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (1.50s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (1.89s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (1.22s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (1.12s) --- PASS: TestQueryShardingCorrectness/stddev() (1.60s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest (0.29s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.06s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.06s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.05s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.03s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest (1.31s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.27s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.33s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.23s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.29s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2 (6.66s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest (0.82s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.08s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.11s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.30s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.20s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest (5.84s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (1.23s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (1.73s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.63s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (1.55s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le (18.25s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest (3.84s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.86s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.63s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.57s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (1.29s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest (14.39s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (2.25s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (3.51s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (2.25s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (2.51s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without' (3.17s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest (1.27s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.30s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.08s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.11s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.70s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest (1.90s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.77s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.43s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.15s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.23s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by' (1.12s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest (0.36s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.04s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.06s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.21s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest (0.75s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.15s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.13s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.17s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.23s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter)) (2.06s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest (0.44s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.04s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.11s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.08s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.15s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest (1.62s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.20s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.15s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.56s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.42s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand (1.44s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest (0.51s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.08s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.11s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.22s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest (0.92s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.15s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.14s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.17s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.29s) --- PASS: TestQueryShardingCorrectness/topk() (2.82s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest (1.16s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.30s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.27s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.07s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.05s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest (1.65s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.28s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.27s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.52s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.37s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter)) (11.07s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest (1.26s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.18s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.20s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.33s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.39s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest (9.80s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (2.01s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (1.41s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (2.06s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (1.51s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time()) (5.05s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest (0.81s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.10s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.13s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.17s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.32s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest (4.24s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.42s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.68s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.83s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (1.93s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset (2.06s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest (0.78s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.09s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.09s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.18s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.37s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest (1.27s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.21s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.21s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.30s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.33s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month() (0.13s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest (0.05s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest (0.07s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric) (0.42s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest (0.12s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest (0.29s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.03s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.06s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.13s) --- PASS: TestQueryShardingCorrectness/sum(metric_native_histogram) (8.65s) --- PASS: TestQueryShardingCorrectness/sum(metric_native_histogram)/*querymiddleware.PrometheusInstantQueryRequest (1.05s) --- PASS: TestQueryShardingCorrectness/sum(metric_native_histogram)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.17s) --- PASS: TestQueryShardingCorrectness/sum(metric_native_histogram)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.17s) --- PASS: TestQueryShardingCorrectness/sum(metric_native_histogram)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.22s) --- PASS: TestQueryShardingCorrectness/sum(metric_native_histogram)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.30s) --- PASS: TestQueryShardingCorrectness/sum(metric_native_histogram)/*querymiddleware.PrometheusRangeQueryRequest (7.60s) --- PASS: TestQueryShardingCorrectness/sum(metric_native_histogram)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (1.58s) --- PASS: TestQueryShardingCorrectness/sum(metric_native_histogram)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (1.50s) --- PASS: TestQueryShardingCorrectness/sum(metric_native_histogram)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (1.09s) --- PASS: TestQueryShardingCorrectness/sum(metric_native_histogram)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (1.43s) --- PASS: TestQueryShardingCorrectness/time() (0.14s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest (0.07s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest (0.06s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.03s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping (3.00s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest (1.12s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.25s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.09s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.21s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.43s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest (1.88s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.36s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.39s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.39s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.51s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series (17.33s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest (0.85s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.18s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.12s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.13s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.30s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest (16.49s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (3.50s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (3.87s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (2.63s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (4.09s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar (48.55s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest (2.64s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.43s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.87s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.63s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.61s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest (45.91s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (13.88s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (12.91s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (7.93s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (8.80s) --- PASS: TestQueryShardingCorrectness/histogram_sum(sum(metric_native_histogram)) (10.51s) --- PASS: TestQueryShardingCorrectness/histogram_sum(sum(metric_native_histogram))/*querymiddleware.PrometheusInstantQueryRequest (0.93s) --- PASS: TestQueryShardingCorrectness/histogram_sum(sum(metric_native_histogram))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.07s) --- PASS: TestQueryShardingCorrectness/histogram_sum(sum(metric_native_histogram))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.14s) --- PASS: TestQueryShardingCorrectness/histogram_sum(sum(metric_native_histogram))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.17s) --- PASS: TestQueryShardingCorrectness/histogram_sum(sum(metric_native_histogram))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.47s) --- PASS: TestQueryShardingCorrectness/histogram_sum(sum(metric_native_histogram))/*querymiddleware.PrometheusRangeQueryRequest (9.58s) --- PASS: TestQueryShardingCorrectness/histogram_sum(sum(metric_native_histogram))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (2.03s) --- PASS: TestQueryShardingCorrectness/histogram_sum(sum(metric_native_histogram))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (1.17s) --- PASS: TestQueryShardingCorrectness/histogram_sum(sum(metric_native_histogram))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (1.15s) --- PASS: TestQueryShardingCorrectness/histogram_sum(sum(metric_native_histogram))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (1.85s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without' (19.30s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest (3.54s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.81s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.72s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.62s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.90s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest (15.76s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (4.06s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (2.24s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (2.17s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (2.02s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1) (1.95s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest (0.46s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.06s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.05s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.09s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.21s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest (1.48s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.19s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.20s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.31s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.62s) --- PASS: TestQueryShardingCorrectness/triple_subquery (38.70s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest (6.80s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (1.04s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (1.44s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (1.59s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (1.98s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest (31.90s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (8.09s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (6.82s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (7.22s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (6.96s) --- PASS: TestQueryShardingCorrectness/subquery_max (31.74s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest (3.23s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (1.34s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.42s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.53s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.63s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest (28.52s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (6.88s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (6.64s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (6.03s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (8.57s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without' (2.27s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest (0.35s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.07s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.07s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.16s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest (1.91s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.21s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.99s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.44s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.22s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping (1.79s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest (0.31s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.05s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.05s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.06s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.12s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest (1.48s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.05s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.45s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.65s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.20s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without' (8.14s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest (1.78s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.19s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.45s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.44s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.62s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest (6.36s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.92s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.97s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (2.24s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (1.97s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1 (2.31s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest (0.52s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.07s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.11s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.12s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.17s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest (1.79s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.38s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.23s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.23s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.29s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset (1.89s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest (0.84s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.10s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.13s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.24s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.30s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest (1.04s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.12s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.16s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.24s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.45s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_avg (0.97s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest (0.96s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.17s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.25s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.14s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.19s) --- PASS: TestQueryShardingCorrectness/label_replace (1.70s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest (0.53s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.06s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.09s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.08s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.19s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest (1.18s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.18s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.19s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.25s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.32s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation) (1.54s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest (0.46s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.04s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.09s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.09s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.21s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest (1.08s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.16s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.22s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.26s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.28s) --- PASS: TestQueryShardingCorrectness/nested_count() (3.81s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest (0.60s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.09s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.12s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.18s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.17s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest (3.21s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.53s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.78s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.71s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.89s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by' (3.07s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest (1.02s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.13s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.10s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.31s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.40s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest (2.05s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.35s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.32s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.49s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.72s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping (0.82s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest (0.31s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.04s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.06s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.13s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest (0.51s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.04s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.09s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.15s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.21s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping (0.40s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest (0.18s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.03s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.02s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.03s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest (0.21s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.04s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.04s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.04s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.04s) --- PASS: TestQueryShardingCorrectness/histogram_fraction(0,_0.5,_sum(metric_native_histogram)) (8.55s) --- PASS: TestQueryShardingCorrectness/histogram_fraction(0,_0.5,_sum(metric_native_histogram))/*querymiddleware.PrometheusInstantQueryRequest (0.73s) --- PASS: TestQueryShardingCorrectness/histogram_fraction(0,_0.5,_sum(metric_native_histogram))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.13s) --- PASS: TestQueryShardingCorrectness/histogram_fraction(0,_0.5,_sum(metric_native_histogram))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.16s) --- PASS: TestQueryShardingCorrectness/histogram_fraction(0,_0.5,_sum(metric_native_histogram))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.12s) --- PASS: TestQueryShardingCorrectness/histogram_fraction(0,_0.5,_sum(metric_native_histogram))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.22s) --- PASS: TestQueryShardingCorrectness/histogram_fraction(0,_0.5,_sum(metric_native_histogram))/*querymiddleware.PrometheusRangeQueryRequest (7.80s) --- PASS: TestQueryShardingCorrectness/histogram_fraction(0,_0.5,_sum(metric_native_histogram))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (1.19s) --- PASS: TestQueryShardingCorrectness/histogram_fraction(0,_0.5,_sum(metric_native_histogram))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.97s) --- PASS: TestQueryShardingCorrectness/histogram_fraction(0,_0.5,_sum(metric_native_histogram))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (1.34s) --- PASS: TestQueryShardingCorrectness/histogram_fraction(0,_0.5,_sum(metric_native_histogram))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (1.91s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order (7.78s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest (1.76s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.20s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.35s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.33s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.67s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest (6.01s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.89s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.84s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.96s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (2.46s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr (2.57s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest (0.86s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.18s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.18s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.12s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.19s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest (1.71s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.35s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.37s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.28s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.32s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month() (2.47s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest (0.62s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.09s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.09s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.17s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.23s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest (1.85s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.38s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.27s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.27s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.40s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time()) (5.53s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest (0.66s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.09s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.10s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.15s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.24s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest (4.87s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.75s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.59s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (1.73s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (1.17s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping (1.75s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest (0.65s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.06s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.12s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.15s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.24s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest (1.09s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.20s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.22s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.19s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.23s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by' (3.56s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest (0.74s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.11s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.11s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.16s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.29s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest (2.82s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.40s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.60s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.60s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.77s) --- PASS: TestQueryShardingCorrectness/sum()_offset (3.63s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest (0.70s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.06s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.10s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.16s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.27s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest (2.93s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*