2023-04-03 05:56:53,659 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-04-03 05:56:53,981 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-04-03 05:56:54,088 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-04-03 05:56:54,143 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-03 05:56:54,150 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-04-03 05:56:54,162 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=372, ProcessCount=169, AvailableMemoryMB=4480
2023-04-03 05:56:54,164 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-03 05:56:54,165 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-04-03 05:56:54,282 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1033591768:84 , srv106969010:84 , srv1078388012:84 , srv1098502486:84 , srv11063731:84 , srv1114652559:84 , srv1140585584:84 , srv1149891174:84 , srv115924299:84 , srv1162221144:84 , srv1201493646:84 , srv1231510737:84 , srv1339419559:84 , srv1354644316:84 , srv1376241191:84 , srv1402014031:84 , srv1433126291:84 , srv1436528708:84 , srv1497071834:84 , srv1521342796:84 , srv1530339707:84 , srv1534030632:84 , srv1542845926:84 , srv1559040312:84 , srv1566617444:84 , srv1585138117:84 , srv1597265753:84 , srv1597317314:84 , srv1608383772:84 , srv1627480782:84 , srv1633200257:84 , srv1638457566:84 , srv1651541553:84 , srv1652328953:84 , srv1653771996:84 , srv165564673:84 , srv1663312954:84 , srv1663695123:84 , srv1675109808:84 , srv167922542:84 , srv1685012155:84 , srv1691488905:84 , srv1692188137:84 , srv1718829453:84 , srv1721138823:84 , srv1737072198:84 , srv1740161031:84 , srv1740544989:84 , srv1757950401:84 , srv1767356405:84 , srv177661154:84 , srv1780726828:84 , srv178259435:84 , srv1825314680:84 , srv1833355389:84 , srv1842965388:84 , srv1943219368:84 , srv1953952025:84 , srv197394754:84 , srv1987101550:84 , srv1988157177:84 , srv2024063215:84 , srv2084724219:84 , srv2103563055:84 , srv2112910106:84 , srv230465192:84 , srv285652991:84 , srv328637205:84 , srv332739295:84 , srv344095741:84 , srv387854149:84 , srv39269570:84 , srv446150368:84 , srv48062264:84 , srv482574146:84 , srv514126314:84 , srv530627249:84 , srv549843538:84 , srv59022:84 , srv59306228:84 , srv613257623:84 , srv640590711:84 , srv667320895:84 , srv674849397:84 , srv678805373:84 , srv717920556:84 , srv740328276:84 , srv773815450:84 , srv774567427:84 , srv808734718:84 , srv835093852:84 , srv86087377:84 , srv867652848:84 , srv895555034:84 , srv955079004:84 , srv957314930:84 , srv972935506:84 , srv97586790:84 , srv995828041:84 , srv58714777:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-04-03 05:56:54,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-03 05:56:54,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv344095741=69, srv1721138823=44, srv549843538=77, srv106969010=1, srv1402014031=15, srv1833355389=54, srv178259435=52, srv446150368=72, srv717920556=86, srv167922542=39, srv1652328953=33, srv1608383772=28, srv808734718=90, srv773815450=88, srv1767356405=49, srv640590711=82, srv740328276=87, srv1651541553=32, srv1114652559=5, srv1718829453=43, srv1376241191=14, srv1559040312=23, srv867652848=93, srv1691488905=41, srv972935506=97, srv1339419559=12, srv1685012155=40, srv2024063215=61, srv2112910106=64, srv58714777=78, srv1078388012=2, srv1740544989=47, srv1988157177=60, srv1825314680=53, srv1597317314=27, srv230465192=65, srv995828041=99, srv1597265753=26, srv1842965388=55, srv97586790=98, srv1033591768=0, srv674849397=84, srv1534030632=21, srv1530339707=20, srv835093852=91, srv48062264=73, srv387854149=70, srv2103563055=63, srv1780726828=51, srv1740161031=46, srv1140585584=6, srv1675109808=38, srv1231510737=11, srv115924299=8, srv39269570=71, srv1585138117=25, srv1737072198=45, srv774567427=89, srv1627480782=29, srv895555034=94, srv530627249=76, srv1433126291=16, srv955079004=95, srv1497071834=18, srv332739295=68, srv328637205=67, srv1633200257=30, srv2084724219=62, srv59022=79, srv514126314=75, srv1692188137=42, srv1987101550=59, srv285652991=66, srv1149891174=7, srv678805373=85, srv1653771996=34, srv1201493646=10, srv1663312954=36, srv1354644316=13, srv482574146=74, srv59306228=80, srv613257623=81, srv1162221144=9, srv165564673=35, srv177661154=50, srv86087377=92, srv1943219368=56, srv1757950401=48, srv667320895=83, srv1098502486=3, srv1663695123=37, srv957314930=96, srv1542845926=22, srv1436528708=17, srv1566617444=24, srv197394754=58, srv1953952025=57, srv1638457566=31, srv1521342796=19, srv11063731=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-03 05:56:54,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-03 05:56:54,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-03 05:56:54,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-03 05:56:54,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-03 05:56:54,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-03 05:56:54,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-03 05:56:54,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-03 05:56:54,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-03 05:56:54,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-03 05:56:54,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-03 05:56:54,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-03 05:56:54,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-03 05:56:54,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-03 05:56:54,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-03 05:56:54,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-03 05:56:54,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-03 05:56:54,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-03 05:56:54,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-03 05:56:54,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-03 05:56:54,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-03 05:56:54,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-03 05:56:54,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-03 05:56:54,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-03 05:56:54,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-03 05:56:54,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-03 05:56:54,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-03 05:56:54,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-03 05:56:54,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-03 05:56:54,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-03 05:56:54,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-03 05:56:54,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-03 05:56:54,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-03 05:56:54,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-03 05:56:54,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-03 05:56:54,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-03 05:56:54,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-03 05:56:54,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-03 05:56:54,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-03 05:56:54,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-03 05:56:54,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-03 05:56:54,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-03 05:56:54,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-03 05:56:54,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-03 05:56:54,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-03 05:56:54,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-03 05:56:54,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-03 05:56:54,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-03 05:56:54,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-03 05:56:54,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-03 05:56:54,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-03 05:56:54,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-03 05:56:54,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-03 05:56:54,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-03 05:56:54,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-03 05:56:54,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-03 05:56:54,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-03 05:56:54,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-03 05:56:54,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-03 05:56:54,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-03 05:56:54,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-03 05:56:54,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-03 05:56:54,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-03 05:56:54,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-03 05:56:54,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-03 05:56:54,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-03 05:56:54,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-03 05:56:54,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-03 05:56:54,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-03 05:56:54,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-03 05:56:54,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-03 05:56:54,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-03 05:56:54,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-03 05:56:54,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-03 05:56:54,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-03 05:56:54,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-03 05:56:54,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-03 05:56:54,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-03 05:56:54,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-03 05:56:54,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-03 05:56:54,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-03 05:56:54,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-03 05:56:54,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-03 05:56:54,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-03 05:56:54,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-03 05:56:54,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-03 05:56:54,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-03 05:56:54,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-03 05:56:54,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-03 05:56:54,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-03 05:56:54,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-03 05:56:54,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-03 05:56:54,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-03 05:56:54,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-03 05:56:54,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-03 05:56:54,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-03 05:56:54,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-03 05:56:54,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-03 05:56:54,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-03 05:56:54,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-03 05:56:54,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-03 05:56:54,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-03 05:56:54,639 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-03 05:56:54,639 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-03 05:56:54,639 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-03 05:56:55,279 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 873 ms to try 5000 different iterations.  Found a solution that moves 4639 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18713475909280555. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04039461169446831, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05063014810115836, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5154444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04039461169446831, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18258333333333332, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24775, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-03 05:56:55,301 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1627480782:81 , srv972935506:81 , srv1098502486:82 , srv1652328953:82 , srv1436528708:83 , srv1521342796:83 , srv1559040312:83 , srv1651541553:83 , srv1663695123:83 , srv1691488905:83 , srv1740161031:83 , srv59306228:83 , srv955079004:83 , srv1114652559:84 , srv1534030632:84 , srv332739295:84 , srv613257623:84 , srv678805373:84 , srv717920556:84 , srv895555034:84 , srv1149891174:85 , srv1201493646:85 , srv1339419559:85 , srv1530339707:85 , srv1566617444:85 , srv1597317314:85 , srv1685012155:85 , srv1740544989:85 , srv177661154:85 , srv1825314680:85 , srv1842965388:85 , srv1943219368:85 , srv2112910106:85 , srv328637205:85 , srv808734718:85 , srv835093852:85 , srv867652848:85 , srv995828041:85 , srv1162221144:86 , srv1402014031:86 , srv1433126291:86 , srv1497071834:86 , srv1585138117:86 , srv1608383772:86 , srv1663312954:86 , srv1718829453:86 , srv1953952025:86 , srv197394754:86 , srv1988157177:86 , srv2084724219:86 , srv230465192:86 , srv285652991:86 , srv482574146:86 , srv514126314:86 , srv59022:86 , srv773815450:86 , srv957314930:86 , srv1033591768:87 , srv106969010:87 , srv1078388012:87 , srv1231510737:87 , srv1354644316:87 , srv1376241191:87 , srv1633200257:87 , srv165564673:87 , srv1675109808:87 , srv167922542:87 , srv1737072198:87 , srv1833355389:87 , srv39269570:87 , srv640590711:87 , srv774567427:87 , srv86087377:87 , srv11063731:88 , srv1542845926:88 , srv1597265753:88 , srv1692188137:88 , srv1721138823:88 , srv1767356405:88 , srv1780726828:88 , srv178259435:88 , srv1638457566:89 , srv1987101550:89 , srv48062264:89 , srv667320895:89 , srv674849397:89 , srv115924299:90 , srv2024063215:90 , srv2103563055:90 , srv387854149:90 , srv530627249:90 , srv549843538:90 , srv740328276:90 , srv344095741:91 , srv446150368:91 , srv97586790:92 , srv1653771996:93 , srv1757950401:93 , srv1140585584:95 , srv58714777:449 }
2023-04-03 05:56:55,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-03 05:56:55,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv344095741=69, srv1721138823=44, srv549843538=77, srv106969010=1, srv1402014031=15, srv1833355389=54, srv178259435=52, srv446150368=72, srv717920556=86, srv167922542=39, srv1652328953=33, srv1608383772=28, srv808734718=90, srv773815450=88, srv1767356405=49, srv640590711=82, srv740328276=87, srv1651541553=32, srv1114652559=5, srv1718829453=43, srv1376241191=14, srv1559040312=23, srv867652848=93, srv1691488905=41, srv972935506=97, srv1339419559=12, srv1685012155=40, srv2024063215=61, srv2112910106=64, srv58714777=78, srv1078388012=2, srv1740544989=47, srv1988157177=60, srv1825314680=53, srv1597317314=27, srv230465192=65, srv995828041=99, srv1597265753=26, srv1842965388=55, srv97586790=98, srv1033591768=0, srv674849397=84, srv1534030632=21, srv1530339707=20, srv835093852=91, srv48062264=73, srv387854149=70, srv2103563055=63, srv1780726828=51, srv1740161031=46, srv1140585584=6, srv1675109808=38, srv1231510737=11, srv115924299=8, srv39269570=71, srv1585138117=25, srv1737072198=45, srv774567427=89, srv1627480782=29, srv895555034=94, srv530627249=76, srv1433126291=16, srv955079004=95, srv1497071834=18, srv332739295=68, srv328637205=67, srv1633200257=30, srv2084724219=62, srv59022=79, srv514126314=75, srv1692188137=42, srv1987101550=59, srv285652991=66, srv1149891174=7, srv678805373=85, srv1653771996=34, srv1201493646=10, srv1663312954=36, srv1354644316=13, srv482574146=74, srv59306228=80, srv613257623=81, srv1162221144=9, srv165564673=35, srv177661154=50, srv86087377=92, srv1943219368=56, srv1757950401=48, srv667320895=83, srv1098502486=3, srv1663695123=37, srv957314930=96, srv1542845926=22, srv1436528708=17, srv1566617444=24, srv197394754=58, srv1953952025=57, srv1638457566=31, srv1521342796=19, srv11063731=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-03 05:56:55,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-03 05:56:55,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-03 05:56:55,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-03 05:56:55,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-03 05:56:55,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-03 05:56:55,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-03 05:56:55,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-03 05:56:55,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-03 05:56:55,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-03 05:56:55,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-03 05:56:55,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-03 05:56:55,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-03 05:56:55,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-03 05:56:55,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-03 05:56:55,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-03 05:56:55,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-03 05:56:55,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-03 05:56:55,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-03 05:56:55,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-03 05:56:55,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-03 05:56:55,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-03 05:56:55,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-03 05:56:55,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-03 05:56:55,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-03 05:56:55,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-03 05:56:55,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-03 05:56:55,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-03 05:56:55,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-03 05:56:55,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-03 05:56:55,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-03 05:56:55,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-03 05:56:55,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-03 05:56:55,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-03 05:56:55,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-03 05:56:55,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-03 05:56:55,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-03 05:56:55,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-03 05:56:55,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-03 05:56:55,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-03 05:56:55,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-03 05:56:55,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-03 05:56:55,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-03 05:56:55,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-03 05:56:55,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-03 05:56:55,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-03 05:56:55,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-03 05:56:55,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-03 05:56:55,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-03 05:56:55,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-03 05:56:55,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-03 05:56:55,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-03 05:56:55,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-03 05:56:55,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-03 05:56:55,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-03 05:56:55,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-03 05:56:55,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-03 05:56:55,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-03 05:56:55,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-03 05:56:55,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-03 05:56:55,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-03 05:56:55,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-03 05:56:55,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-03 05:56:55,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-03 05:56:55,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-03 05:56:55,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-03 05:56:55,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-03 05:56:55,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-03 05:56:55,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-03 05:56:55,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-03 05:56:55,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-03 05:56:55,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-03 05:56:55,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-03 05:56:55,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-03 05:56:55,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-03 05:56:55,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-03 05:56:55,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-03 05:56:55,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-03 05:56:55,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-03 05:56:55,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-03 05:56:55,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-03 05:56:55,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-03 05:56:55,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-03 05:56:55,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-03 05:56:55,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-03 05:56:55,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-03 05:56:55,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-03 05:56:55,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-03 05:56:55,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-03 05:56:55,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-03 05:56:55,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-03 05:56:55,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-03 05:56:55,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-03 05:56:55,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-03 05:56:55,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-03 05:56:55,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-03 05:56:55,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-03 05:56:55,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-03 05:56:55,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-03 05:56:55,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-03 05:56:55,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-03 05:56:55,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-03 05:56:55,506 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04039461169446831, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05063014810115836, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04039461169446831, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18258333333333332, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24775, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-03 05:56:55,506 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-03 05:56:55,506 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18710227612053307, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04039461169446831, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05063014810115836, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04039461169446831, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18258333333333332, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24775, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-03 05:56:55,791 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 441 ms to try 5000 different iterations.  Found a solution that moves 2699 regions; Going from a computed imbalance of 0.18710227612053307 to a new imbalance of 0.015056186390700863. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022627064279529586); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03304929488288506, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29988888888888887, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022627064279529586); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.045, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-03 05:56:55,799 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1140585584:77 , srv1653771996:78 , srv97586790:78 , srv1757950401:79 , srv446150368:80 , srv549843538:80 , srv387854149:81 , srv1638457566:82 , srv2024063215:82 , srv740328276:82 , srv115924299:83 , srv1354644316:83 , srv1433126291:83 , srv165564673:83 , srv167922542:83 , srv1767356405:83 , srv344095741:83 , srv1033591768:84 , srv1231510737:84 , srv1376241191:84 , srv1597265753:84 , srv1692188137:84 , srv1737072198:84 , srv1987101550:84 , srv328637205:84 , srv530627249:84 , srv774567427:84 , srv106969010:85 , srv1078388012:85 , srv11063731:85 , srv1542845926:85 , srv1566617444:85 , srv1597317314:85 , srv1633200257:85 , srv1663312954:85 , srv1721138823:85 , srv177661154:85 , srv1780726828:85 , srv1833355389:85 , srv197394754:85 , srv2103563055:85 , srv285652991:85 , srv482574146:85 , srv667320895:85 , srv674849397:85 , srv835093852:85 , srv995828041:85 , srv1201493646:86 , srv1530339707:86 , srv1585138117:86 , srv1608383772:86 , srv1675109808:86 , srv1842965388:86 , srv1943219368:86 , srv1988157177:86 , srv2112910106:86 , srv48062264:86 , srv59022:86 , srv640590711:86 , srv678805373:86 , srv773815450:86 , srv808734718:86 , srv1114652559:87 , srv1162221144:87 , srv1402014031:87 , srv1497071834:87 , srv1691488905:87 , srv1740161031:87 , srv1740544989:87 , srv178259435:87 , srv1825314680:87 , srv1953952025:87 , srv2084724219:87 , srv230465192:87 , srv332739295:87 , srv867652848:87 , srv1149891174:88 , srv1339419559:88 , srv1436528708:88 , srv1534030632:88 , srv1651541553:88 , srv1652328953:88 , srv1685012155:88 , srv39269570:88 , srv613257623:88 , srv717920556:88 , srv895555034:88 , srv957314930:88 , srv1098502486:89 , srv1521342796:89 , srv1663695123:89 , srv1627480782:90 , srv1718829453:90 , srv514126314:90 , srv59306228:90 , srv955079004:90 , srv1559040312:91 , srv86087377:91 , srv972935506:91 , srv58714777:526 }
2023-04-03 05:56:55,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-03 05:56:55,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv344095741=69, srv1721138823=44, srv549843538=77, srv106969010=1, srv1402014031=15, srv1833355389=54, srv178259435=52, srv446150368=72, srv717920556=86, srv167922542=39, srv1652328953=33, srv1608383772=28, srv808734718=90, srv773815450=88, srv1767356405=49, srv640590711=82, srv740328276=87, srv1651541553=32, srv1114652559=5, srv1718829453=43, srv1376241191=14, srv1559040312=23, srv867652848=93, srv1691488905=41, srv972935506=97, srv1339419559=12, srv1685012155=40, srv2024063215=61, srv2112910106=64, srv58714777=78, srv1078388012=2, srv1740544989=47, srv1988157177=60, srv1825314680=53, srv1597317314=27, srv230465192=65, srv995828041=99, srv1597265753=26, srv1842965388=55, srv97586790=98, srv1033591768=0, srv674849397=84, srv1534030632=21, srv1530339707=20, srv835093852=91, srv48062264=73, srv387854149=70, srv2103563055=63, srv1780726828=51, srv1740161031=46, srv1140585584=6, srv1675109808=38, srv1231510737=11, srv115924299=8, srv39269570=71, srv1585138117=25, srv1737072198=45, srv774567427=89, srv1627480782=29, srv895555034=94, srv530627249=76, srv1433126291=16, srv955079004=95, srv1497071834=18, srv332739295=68, srv328637205=67, srv1633200257=30, srv2084724219=62, srv59022=79, srv514126314=75, srv1692188137=42, srv1987101550=59, srv285652991=66, srv1149891174=7, srv678805373=85, srv1653771996=34, srv1201493646=10, srv1663312954=36, srv1354644316=13, srv482574146=74, srv59306228=80, srv613257623=81, srv1162221144=9, srv165564673=35, srv177661154=50, srv86087377=92, srv1943219368=56, srv1757950401=48, srv667320895=83, srv1098502486=3, srv1663695123=37, srv957314930=96, srv1542845926=22, srv1436528708=17, srv1566617444=24, srv197394754=58, srv1953952025=57, srv1638457566=31, srv1521342796=19, srv11063731=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-03 05:56:55,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-03 05:56:55,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-03 05:56:55,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-03 05:56:55,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-03 05:56:55,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-03 05:56:55,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-03 05:56:55,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-03 05:56:55,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-03 05:56:55,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-03 05:56:55,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-03 05:56:55,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-03 05:56:55,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-03 05:56:55,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-03 05:56:55,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-03 05:56:55,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-03 05:56:55,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-03 05:56:55,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-03 05:56:55,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-03 05:56:55,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-03 05:56:55,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-03 05:56:55,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-03 05:56:55,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-03 05:56:55,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-03 05:56:55,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-03 05:56:55,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-03 05:56:55,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-03 05:56:55,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-03 05:56:55,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-03 05:56:55,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-03 05:56:55,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-03 05:56:55,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-03 05:56:55,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-03 05:56:55,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-03 05:56:55,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-03 05:56:55,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-03 05:56:55,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-03 05:56:55,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-03 05:56:55,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-03 05:56:55,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-03 05:56:55,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-03 05:56:55,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-03 05:56:55,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-03 05:56:55,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-03 05:56:55,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-03 05:56:55,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-03 05:56:55,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-03 05:56:55,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-03 05:56:55,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-03 05:56:55,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-03 05:56:55,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-03 05:56:55,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-03 05:56:55,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-03 05:56:55,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-03 05:56:55,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-03 05:56:55,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-03 05:56:55,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-03 05:56:55,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-03 05:56:55,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-03 05:56:55,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-03 05:56:55,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-03 05:56:55,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-03 05:56:55,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-03 05:56:55,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-03 05:56:55,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-03 05:56:55,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-03 05:56:55,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-03 05:56:55,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-03 05:56:55,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-03 05:56:55,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-03 05:56:55,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-03 05:56:55,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-03 05:56:55,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-03 05:56:55,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-03 05:56:55,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-03 05:56:55,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-03 05:56:55,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-03 05:56:55,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-03 05:56:55,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-03 05:56:55,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-03 05:56:55,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-03 05:56:55,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-03 05:56:55,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-03 05:56:55,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-03 05:56:55,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-03 05:56:55,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-03 05:56:55,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-03 05:56:55,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-03 05:56:55,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-03 05:56:55,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-03 05:56:55,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-03 05:56:55,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-03 05:56:55,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-03 05:56:55,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-03 05:56:55,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-03 05:56:55,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-03 05:56:55,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-03 05:56:55,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-03 05:56:55,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-03 05:56:55,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-03 05:56:55,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-03 05:56:55,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-03 05:56:55,943 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022627064279529586); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03304929488288506, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022627064279529586); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.045, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-03 05:56:55,944 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-03 05:56:55,944 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015037287588768669, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022627064279529586); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03304929488288506, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022627064279529586); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.045, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-03 05:56:56,164 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 315 ms to try 5000 different iterations.  Found a solution that moves 982 regions; Going from a computed imbalance of 0.015037287588768669 to a new imbalance of 7.698872044919356E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00622759107799657); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007842444017689564); MoveCostFunction : (multiplier=7.0, imbalance=0.10911111111111112, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00622759107799657); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-03 05:56:56,166 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv86087377:80 , srv1718829453:81 , srv178259435:82 , srv48062264:82 , srv1559040312:83 , srv2103563055:83 , srv39269570:83 , srv667320895:83 , srv115924299:84 , srv1162221144:84 , srv1402014031:84 , srv1542845926:84 , srv1663695123:84 , srv1675109808:84 , srv1685012155:84 , srv1953952025:84 , srv1987101550:84 , srv2084724219:84 , srv230465192:84 , srv344095741:84 , srv514126314:84 , srv640590711:84 , srv674849397:84 , srv955079004:84 , srv1033591768:85 , srv11063731:85 , srv1140585584:85 , srv1149891174:85 , srv1339419559:85 , srv1585138117:85 , srv1597265753:85 , srv1608383772:85 , srv1692188137:85 , srv1721138823:85 , srv1780726828:85 , srv1825314680:85 , srv1833355389:85 , srv197394754:85 , srv530627249:85 , srv59306228:85 , srv613257623:85 , srv717920556:85 , srv773815450:85 , srv867652848:85 , srv895555034:85 , srv957314930:85 , srv972935506:85 , srv106969010:86 , srv1078388012:86 , srv1098502486:86 , srv1114652559:86 , srv1201493646:86 , srv1354644316:86 , srv1376241191:86 , srv1497071834:86 , srv1530339707:86 , srv1627480782:86 , srv1633200257:86 , srv1638457566:86 , srv1651541553:86 , srv1663312954:86 , srv1737072198:86 , srv1740544989:86 , srv1757950401:86 , srv1767356405:86 , srv1842965388:86 , srv1943219368:86 , srv1988157177:86 , srv2024063215:86 , srv285652991:86 , srv446150368:86 , srv482574146:86 , srv59022:86 , srv740328276:86 , srv774567427:86 , srv808734718:86 , srv1231510737:87 , srv1436528708:87 , srv1521342796:87 , srv1534030632:87 , srv1566617444:87 , srv1597317314:87 , srv1653771996:87 , srv1691488905:87 , srv1740161031:87 , srv177661154:87 , srv2112910106:87 , srv332739295:87 , srv387854149:87 , srv549843538:87 , srv97586790:87 , srv1433126291:88 , srv1652328953:88 , srv165564673:88 , srv678805373:88 , srv835093852:88 , srv995828041:88 , srv328637205:89 , srv167922542:90 , srv58714777:538 }
2023-04-03 05:56:56,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-03 05:56:56,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv344095741=69, srv1721138823=44, srv549843538=77, srv106969010=1, srv1402014031=15, srv1833355389=54, srv178259435=52, srv446150368=72, srv717920556=86, srv167922542=39, srv1652328953=33, srv1608383772=28, srv808734718=90, srv773815450=88, srv1767356405=49, srv640590711=82, srv740328276=87, srv1651541553=32, srv1114652559=5, srv1718829453=43, srv1376241191=14, srv1559040312=23, srv867652848=93, srv1691488905=41, srv972935506=97, srv1339419559=12, srv1685012155=40, srv2024063215=61, srv2112910106=64, srv58714777=78, srv1078388012=2, srv1740544989=47, srv1988157177=60, srv1825314680=53, srv1597317314=27, srv230465192=65, srv995828041=99, srv1597265753=26, srv1842965388=55, srv97586790=98, srv1033591768=0, srv674849397=84, srv1534030632=21, srv1530339707=20, srv835093852=91, srv48062264=73, srv387854149=70, srv2103563055=63, srv1780726828=51, srv1740161031=46, srv1140585584=6, srv1675109808=38, srv1231510737=11, srv115924299=8, srv39269570=71, srv1585138117=25, srv1737072198=45, srv774567427=89, srv1627480782=29, srv895555034=94, srv530627249=76, srv1433126291=16, srv955079004=95, srv1497071834=18, srv332739295=68, srv328637205=67, srv1633200257=30, srv2084724219=62, srv59022=79, srv514126314=75, srv1692188137=42, srv1987101550=59, srv285652991=66, srv1149891174=7, srv678805373=85, srv1653771996=34, srv1201493646=10, srv1663312954=36, srv1354644316=13, srv482574146=74, srv59306228=80, srv613257623=81, srv1162221144=9, srv165564673=35, srv177661154=50, srv86087377=92, srv1943219368=56, srv1757950401=48, srv667320895=83, srv1098502486=3, srv1663695123=37, srv957314930=96, srv1542845926=22, srv1436528708=17, srv1566617444=24, srv197394754=58, srv1953952025=57, srv1638457566=31, srv1521342796=19, srv11063731=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-03 05:56:56,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-03 05:56:56,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-03 05:56:56,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-03 05:56:56,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-03 05:56:56,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-03 05:56:56,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-03 05:56:56,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-03 05:56:56,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-03 05:56:56,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-03 05:56:56,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-03 05:56:56,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-03 05:56:56,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-03 05:56:56,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-03 05:56:56,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-03 05:56:56,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-03 05:56:56,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-03 05:56:56,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-03 05:56:56,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-03 05:56:56,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-03 05:56:56,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-03 05:56:56,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-03 05:56:56,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-03 05:56:56,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-03 05:56:56,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-03 05:56:56,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-03 05:56:56,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-03 05:56:56,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-03 05:56:56,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-03 05:56:56,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-03 05:56:56,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-03 05:56:56,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-03 05:56:56,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-03 05:56:56,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-03 05:56:56,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-03 05:56:56,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-03 05:56:56,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-03 05:56:56,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-03 05:56:56,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-03 05:56:56,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-03 05:56:56,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-03 05:56:56,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-03 05:56:56,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-03 05:56:56,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-03 05:56:56,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-03 05:56:56,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-03 05:56:56,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-03 05:56:56,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-03 05:56:56,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-03 05:56:56,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-03 05:56:56,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-03 05:56:56,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-03 05:56:56,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-03 05:56:56,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-03 05:56:56,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-03 05:56:56,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-03 05:56:56,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-03 05:56:56,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-03 05:56:56,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-03 05:56:56,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-03 05:56:56,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-03 05:56:56,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-03 05:56:56,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-03 05:56:56,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-03 05:56:56,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-03 05:56:56,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-03 05:56:56,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-03 05:56:56,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-03 05:56:56,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-03 05:56:56,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-03 05:56:56,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-03 05:56:56,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-03 05:56:56,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-03 05:56:56,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-03 05:56:56,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-03 05:56:56,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-03 05:56:56,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-03 05:56:56,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-03 05:56:56,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-03 05:56:56,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-03 05:56:56,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-03 05:56:56,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-03 05:56:56,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-03 05:56:56,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-03 05:56:56,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-03 05:56:56,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-03 05:56:56,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-03 05:56:56,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-03 05:56:56,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-03 05:56:56,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-03 05:56:56,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-03 05:56:56,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-03 05:56:56,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-03 05:56:56,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-03 05:56:56,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-03 05:56:56,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-03 05:56:56,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-03 05:56:56,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-03 05:56:56,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-03 05:56:56,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-03 05:56:56,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-03 05:56:56,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-03 05:56:56,295 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00622759107799657); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007842444017689564); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00622759107799657); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-03 05:56:56,296 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-03 05:56:56,296 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.630110935258691E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00622759107799657); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007842444017689564); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00622759107799657); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-03 05:56:56,611 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 410 ms to try 5000 different iterations.  Found a solution that moves 295 regions; Going from a computed imbalance of 7.630110935258691E-4 to a new imbalance of 2.2093796981905462E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022222222222222222); MoveCostFunction : (multiplier=7.0, imbalance=0.03277777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-03 05:56:56,612 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv167922542:82 , srv1078388012:83 , srv1497071834:83 , srv1521342796:83 , srv1534030632:83 , srv1633200257:83 , srv387854149:83 , srv514126314:83 , srv11063731:84 , srv1149891174:84 , srv1231510737:84 , srv1339419559:84 , srv1376241191:84 , srv1402014031:84 , srv1597265753:84 , srv1652328953:84 , srv1653771996:84 , srv165564673:84 , srv1721138823:84 , srv1740161031:84 , srv1757950401:84 , srv1780726828:84 , srv1988157177:84 , srv2024063215:84 , srv2103563055:84 , srv332739295:84 , srv344095741:84 , srv39269570:84 , srv48062264:84 , srv530627249:84 , srv59306228:84 , srv674849397:84 , srv678805373:84 , srv835093852:84 , srv86087377:84 , srv957314930:84 , srv97586790:84 , srv995828041:84 , srv1033591768:85 , srv106969010:85 , srv1098502486:85 , srv1114652559:85 , srv1140585584:85 , srv115924299:85 , srv1162221144:85 , srv1201493646:85 , srv1433126291:85 , srv1436528708:85 , srv1530339707:85 , srv1542845926:85 , srv1559040312:85 , srv1566617444:85 , srv1585138117:85 , srv1597317314:85 , srv1608383772:85 , srv1627480782:85 , srv1638457566:85 , srv1651541553:85 , srv1663312954:85 , srv1675109808:85 , srv1685012155:85 , srv1691488905:85 , srv1692188137:85 , srv1737072198:85 , srv1740544989:85 , srv1767356405:85 , srv177661154:85 , srv178259435:85 , srv1825314680:85 , srv1833355389:85 , srv1842965388:85 , srv1943219368:85 , srv1953952025:85 , srv197394754:85 , srv1987101550:85 , srv2084724219:85 , srv2112910106:85 , srv230465192:85 , srv285652991:85 , srv328637205:85 , srv446150368:85 , srv482574146:85 , srv549843538:85 , srv59022:85 , srv613257623:85 , srv640590711:85 , srv667320895:85 , srv717920556:85 , srv740328276:85 , srv773815450:85 , srv774567427:85 , srv808734718:85 , srv867652848:85 , srv895555034:85 , srv955079004:85 , srv972935506:85 , srv1354644316:86 , srv1663695123:86 , srv1718829453:86 , srv58714777:629 }
2023-04-03 05:56:56,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-03 05:56:56,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv344095741=69, srv1721138823=44, srv549843538=77, srv106969010=1, srv1402014031=15, srv1833355389=54, srv178259435=52, srv446150368=72, srv717920556=86, srv167922542=39, srv1652328953=33, srv1608383772=28, srv808734718=90, srv773815450=88, srv1767356405=49, srv640590711=82, srv740328276=87, srv1651541553=32, srv1114652559=5, srv1718829453=43, srv1376241191=14, srv1559040312=23, srv867652848=93, srv1691488905=41, srv972935506=97, srv1339419559=12, srv1685012155=40, srv2024063215=61, srv2112910106=64, srv58714777=78, srv1078388012=2, srv1740544989=47, srv1988157177=60, srv1825314680=53, srv1597317314=27, srv230465192=65, srv995828041=99, srv1597265753=26, srv1842965388=55, srv97586790=98, srv1033591768=0, srv674849397=84, srv1534030632=21, srv1530339707=20, srv835093852=91, srv48062264=73, srv387854149=70, srv2103563055=63, srv1780726828=51, srv1740161031=46, srv1140585584=6, srv1675109808=38, srv1231510737=11, srv115924299=8, srv39269570=71, srv1585138117=25, srv1737072198=45, srv774567427=89, srv1627480782=29, srv895555034=94, srv530627249=76, srv1433126291=16, srv955079004=95, srv1497071834=18, srv332739295=68, srv328637205=67, srv1633200257=30, srv2084724219=62, srv59022=79, srv514126314=75, srv1692188137=42, srv1987101550=59, srv285652991=66, srv1149891174=7, srv678805373=85, srv1653771996=34, srv1201493646=10, srv1663312954=36, srv1354644316=13, srv482574146=74, srv59306228=80, srv613257623=81, srv1162221144=9, srv165564673=35, srv177661154=50, srv86087377=92, srv1943219368=56, srv1757950401=48, srv667320895=83, srv1098502486=3, srv1663695123=37, srv957314930=96, srv1542845926=22, srv1436528708=17, srv1566617444=24, srv197394754=58, srv1953952025=57, srv1638457566=31, srv1521342796=19, srv11063731=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-03 05:56:56,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-03 05:56:56,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-03 05:56:56,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-03 05:56:56,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-03 05:56:56,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-03 05:56:56,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-03 05:56:56,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-03 05:56:56,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-03 05:56:56,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-03 05:56:56,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-03 05:56:56,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-03 05:56:56,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-03 05:56:56,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-03 05:56:56,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-03 05:56:56,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-03 05:56:56,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-03 05:56:56,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-03 05:56:56,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-03 05:56:56,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-03 05:56:56,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-03 05:56:56,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-03 05:56:56,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-03 05:56:56,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-03 05:56:56,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-03 05:56:56,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-03 05:56:56,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-03 05:56:56,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-03 05:56:56,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-03 05:56:56,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-03 05:56:56,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-03 05:56:56,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-03 05:56:56,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-03 05:56:56,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-03 05:56:56,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-03 05:56:56,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-03 05:56:56,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-03 05:56:56,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-03 05:56:56,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-03 05:56:56,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-03 05:56:56,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-03 05:56:56,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-03 05:56:56,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-03 05:56:56,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-03 05:56:56,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-03 05:56:56,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-03 05:56:56,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-03 05:56:56,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-03 05:56:56,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-03 05:56:56,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-03 05:56:56,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-03 05:56:56,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-03 05:56:56,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-03 05:56:56,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-03 05:56:56,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-03 05:56:56,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-03 05:56:56,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-03 05:56:56,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-03 05:56:56,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-03 05:56:56,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-03 05:56:56,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-03 05:56:56,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-03 05:56:56,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-03 05:56:56,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-03 05:56:56,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-03 05:56:56,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-03 05:56:56,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-03 05:56:56,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-03 05:56:56,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-03 05:56:56,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-03 05:56:56,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-03 05:56:56,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-03 05:56:56,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-03 05:56:56,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-03 05:56:56,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-03 05:56:56,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-03 05:56:56,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-03 05:56:56,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-03 05:56:56,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-03 05:56:56,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-03 05:56:56,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-03 05:56:56,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-03 05:56:56,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-03 05:56:56,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-03 05:56:56,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-03 05:56:56,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-03 05:56:56,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-03 05:56:56,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-03 05:56:56,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-03 05:56:56,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-03 05:56:56,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-03 05:56:56,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-03 05:56:56,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-03 05:56:56,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-03 05:56:56,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-03 05:56:56,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-03 05:56:56,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-03 05:56:56,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-03 05:56:56,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-03 05:56:56,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-03 05:56:56,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-03 05:56:56,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-03 05:56:56,747 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=2.002816283222151E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022222222222222222); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-03 05:56:56,747 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv167922542:82 , srv1078388012:83 , srv1497071834:83 , srv1521342796:83 , srv1534030632:83 , srv1633200257:83 , srv387854149:83 , srv514126314:83 , srv11063731:84 , srv1149891174:84 , srv1231510737:84 , srv1339419559:84 , srv1376241191:84 , srv1402014031:84 , srv1597265753:84 , srv1652328953:84 , srv1653771996:84 , srv165564673:84 , srv1721138823:84 , srv1740161031:84 , srv1757950401:84 , srv1780726828:84 , srv1988157177:84 , srv2024063215:84 , srv2103563055:84 , srv332739295:84 , srv344095741:84 , srv39269570:84 , srv48062264:84 , srv530627249:84 , srv59306228:84 , srv674849397:84 , srv678805373:84 , srv835093852:84 , srv86087377:84 , srv957314930:84 , srv97586790:84 , srv995828041:84 , srv1033591768:85 , srv106969010:85 , srv1098502486:85 , srv1114652559:85 , srv1140585584:85 , srv115924299:85 , srv1162221144:85 , srv1201493646:85 , srv1433126291:85 , srv1436528708:85 , srv1530339707:85 , srv1542845926:85 , srv1559040312:85 , srv1566617444:85 , srv1585138117:85 , srv1597317314:85 , srv1608383772:85 , srv1627480782:85 , srv1638457566:85 , srv1651541553:85 , srv1663312954:85 , srv1675109808:85 , srv1685012155:85 , srv1691488905:85 , srv1692188137:85 , srv1737072198:85 , srv1740544989:85 , srv1767356405:85 , srv177661154:85 , srv178259435:85 , srv1825314680:85 , srv1833355389:85 , srv1842965388:85 , srv1943219368:85 , srv1953952025:85 , srv197394754:85 , srv1987101550:85 , srv2084724219:85 , srv2112910106:85 , srv230465192:85 , srv285652991:85 , srv328637205:85 , srv446150368:85 , srv482574146:85 , srv549843538:85 , srv59022:85 , srv613257623:85 , srv640590711:85 , srv667320895:85 , srv717920556:85 , srv740328276:85 , srv773815450:85 , srv774567427:85 , srv808734718:85 , srv867652848:85 , srv895555034:85 , srv955079004:85 , srv972935506:85 , srv1354644316:86 , srv1663695123:86 , srv1718829453:86 , srv58714777:629 }
2023-04-03 05:56:56,784 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=382 (was 372) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=3987 (was 4480)
2023-04-03 05:56:56,792 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=382, ProcessCount=169, AvailableMemoryMB=3986
2023-04-03 05:56:56,793 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-03 05:56:56,793 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-04-03 05:56:56,805 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1011253988:84 , srv111018572:84 , srv1178015294:84 , srv1201281976:84 , srv1213520167:84 , srv1224853212:84 , srv1317700983:84 , srv1391531430:84 , srv1499055618:84 , srv1630430777:84 , srv1733275970:84 , srv1844570390:84 , srv1882582097:84 , srv1893486579:84 , srv2098511832:84 , srv2110164438:84 , srv2112273776:84 , srv252622475:84 , srv332454081:84 , srv390987983:84 , srv493465590:84 , srv508725230:84 , srv673414134:84 , srv744681364:84 , srv759648325:84 , srv760474733:84 , srv83037288:84 , srv94740150:84 , srv987972021:84 , srv714803405:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-04-03 05:56:56,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-03 05:56:56,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv332454081=18, srv493465590=20, srv744681364=24, srv1224853212=5, srv508725230=21, srv1893486579=13, srv1178015294=2, srv2112273776=16, srv390987983=19, srv1391531430=7, srv1213520167=4, srv1630430777=9, srv2110164438=15, srv1011253988=0, srv1844570390=11, srv1317700983=6, srv252622475=17, srv673414134=22, srv1882582097=12, srv111018572=1, srv987972021=29, srv1499055618=8, srv1733275970=10, srv1201281976=3, srv759648325=25, srv760474733=26, srv94740150=28, srv83037288=27, srv714803405=23, srv2098511832=14} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-03 05:56:56,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-03 05:56:56,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-03 05:56:56,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-03 05:56:56,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-03 05:56:56,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-03 05:56:56,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-03 05:56:56,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-03 05:56:56,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-03 05:56:56,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-03 05:56:56,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-03 05:56:56,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-03 05:56:56,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-03 05:56:56,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-03 05:56:56,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-03 05:56:56,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-03 05:56:56,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-03 05:56:56,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-03 05:56:56,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-03 05:56:56,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-03 05:56:56,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-03 05:56:56,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-03 05:56:56,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-03 05:56:56,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-03 05:56:56,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-03 05:56:56,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-03 05:56:56,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-03 05:56:56,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-03 05:56:56,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-03 05:56:56,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-03 05:56:56,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-03 05:56:56,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-04-03 05:56:56,847 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1284438472207627, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-03 05:56:56,847 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912597765632796, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1284438472207627, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-04-03 05:58:56,826 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6576796 different iterations.  Found a solution that moves 1847 regions; Going from a computed imbalance of 0.9912597765632796 to a new imbalance of 4.3109901406398434E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6840740740740741, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-03 05:58:56,832 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1011253988:90 , srv111018572:90 , srv1178015294:90 , srv1201281976:90 , srv1213520167:90 , srv1224853212:90 , srv1317700983:90 , srv1391531430:90 , srv1499055618:90 , srv1630430777:90 , srv1733275970:90 , srv1844570390:90 , srv1882582097:90 , srv1893486579:90 , srv2098511832:90 , srv2110164438:90 , srv2112273776:90 , srv252622475:90 , srv332454081:90 , srv390987983:90 , srv493465590:90 , srv508725230:90 , srv673414134:90 , srv714803405:90 , srv744681364:90 , srv759648325:90 , srv760474733:90 , srv83037288:90 , srv94740150:90 , srv987972021:90 }
2023-04-03 05:58:56,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-03 05:58:56,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv332454081=18, srv493465590=20, srv744681364=24, srv1224853212=5, srv508725230=21, srv1893486579=13, srv1178015294=2, srv2112273776=16, srv390987983=19, srv1391531430=7, srv1213520167=4, srv1630430777=9, srv2110164438=15, srv1011253988=0, srv1844570390=11, srv1317700983=6, srv252622475=17, srv673414134=22, srv1882582097=12, srv111018572=1, srv987972021=29, srv1499055618=8, srv1733275970=10, srv1201281976=3, srv759648325=25, srv760474733=26, srv94740150=28, srv83037288=27, srv714803405=23, srv2098511832=14} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-03 05:58:56,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-03 05:58:56,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-03 05:58:56,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-03 05:58:56,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-03 05:58:56,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-03 05:58:56,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-03 05:58:56,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-03 05:58:56,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-03 05:58:56,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-03 05:58:56,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-03 05:58:56,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-03 05:58:56,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-03 05:58:56,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-03 05:58:56,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-03 05:58:56,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-03 05:58:56,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-03 05:58:56,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-03 05:58:56,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-03 05:58:56,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-03 05:58:56,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-03 05:58:56,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-03 05:58:56,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-03 05:58:56,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-03 05:58:56,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-03 05:58:56,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-03 05:58:56,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-03 05:58:56,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-03 05:58:56,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-03 05:58:56,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-03 05:58:56,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-03 05:58:56,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-04-03 05:58:56,849 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-03 05:58:56,849 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1011253988:90 , srv111018572:90 , srv1178015294:90 , srv1201281976:90 , srv1213520167:90 , srv1224853212:90 , srv1317700983:90 , srv1391531430:90 , srv1499055618:90 , srv1630430777:90 , srv1733275970:90 , srv1844570390:90 , srv1882582097:90 , srv1893486579:90 , srv2098511832:90 , srv2110164438:90 , srv2112273776:90 , srv252622475:90 , srv332454081:90 , srv390987983:90 , srv493465590:90 , srv508725230:90 , srv673414134:90 , srv714803405:90 , srv744681364:90 , srv759648325:90 , srv760474733:90 , srv83037288:90 , srv94740150:90 , srv987972021:90 }
2023-04-03 05:58:56,862 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=246 (was 382), ProcessCount=169 (was 169), AvailableMemoryMB=3986 (was 3986)