2023-11-27 14:54:03,714 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-11-27 14:54:04,030 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-11-27 14:54:04,142 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-11-27 14:54:04,217 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-27 14:54:04,224 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-11-27 14:54:04,240 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=300, ProcessCount=170, AvailableMemoryMB=9635
2023-11-27 14:54:04,242 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-27 14:54:04,242 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-11-27 14:54:04,374 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1000856827:84 , srv1002504854:84 , srv1039380747:84 , srv1054617934:84 , srv1112708340:84 , srv1112843246:84 , srv1146389561:84 , srv1153703888:84 , srv1177735838:84 , srv1191283213:84 , srv1232275388:84 , srv1261486521:84 , srv1289099110:84 , srv129712100:84 , srv1328352381:84 , srv1355171598:84 , srv1373759038:84 , srv138281273:84 , srv1392032420:84 , srv1410800255:84 , srv1419409168:84 , srv1443289231:84 , srv1447638751:84 , srv1451424236:84 , srv1454497578:84 , srv149352909:84 , srv1498991057:84 , srv1516053655:84 , srv1518696570:84 , srv1523382489:84 , srv1530540232:84 , srv1562617009:84 , srv1563511613:84 , srv1567495364:84 , srv1593036941:84 , srv1606920511:84 , srv1645796270:84 , srv166365277:84 , srv1691191096:84 , srv170679345:84 , srv1725925292:84 , srv1734138194:84 , srv1756255529:84 , srv1773321675:84 , srv1778463864:84 , srv1783285850:84 , srv1799161503:84 , srv1809988751:84 , srv1906124409:84 , srv1908650428:84 , srv192044803:84 , srv1935276149:84 , srv1947207341:84 , srv1973618242:84 , srv1986419332:84 , srv1998163778:84 , srv2027580705:84 , srv2039211357:84 , srv2040726867:84 , srv2042805583:84 , srv2057879659:84 , srv2098764003:84 , srv2126817073:84 , srv2137980786:84 , srv228321302:84 , srv236177160:84 , srv263976351:84 , srv265284146:84 , srv272044504:84 , srv310332301:84 , srv322823046:84 , srv324635108:84 , srv34418757:84 , srv37651788:84 , srv376824824:84 , srv392855187:84 , srv397746616:84 , srv480227054:84 , srv510788721:84 , srv512330658:84 , srv512414194:84 , srv538012091:84 , srv550977424:84 , srv581316021:84 , srv581683752:84 , srv683021025:84 , srv690507778:84 , srv705357198:84 , srv747350067:84 , srv748961219:84 , srv781321102:84 , srv816588091:84 , srv817231890:84 , srv849069256:84 , srv857467757:84 , srv906622147:84 , srv913276901:84 , srv931179430:84 , srv953731216:84 , srv157693946:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-11-27 14:54:04,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-27 14:54:04,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv129712100=13, srv376824824=75, srv1328352381=14, srv310332301=70, srv1498991057=26, srv2126817073=63, srv849069256=94, srv1783285850=46, srv265284146=68, srv1146389561=6, srv857467757=95, srv1289099110=12, srv1935276149=52, srv1523382489=29, srv510788721=79, srv1518696570=28, srv1054617934=3, srv2137980786=64, srv1447638751=22, srv37651788=74, srv817231890=93, srv166365277=38, srv781321102=91, srv1756255529=43, srv392855187=76, srv581683752=85, srv1191283213=9, srv1778463864=45, srv1232275388=10, srv1451424236=23, srv1454497578=24, srv324635108=72, srv34418757=73, srv705357198=88, srv1443289231=21, srv2040726867=59, srv1000856827=0, srv192044803=51, srv1153703888=7, srv1799161503=47, srv480227054=78, srv149352909=25, srv1355171598=15, srv228321302=65, srv397746616=77, srv1177735838=8, srv816588091=92, srv1809988751=48, srv1039380747=2, srv236177160=66, srv1725925292=41, srv138281273=17, srv322823046=71, srv538012091=82, srv1593036941=35, srv2042805583=60, srv512414194=81, srv2039211357=58, srv1908650428=50, srv1773321675=44, srv1606920511=36, srv913276901=97, srv550977424=83, srv512330658=80, srv1973618242=54, srv683021025=86, srv1567495364=33, srv1002504854=1, srv1410800255=19, srv931179430=98, srv1516053655=27, srv690507778=87, srv1734138194=42, srv2057879659=61, srv1563511613=32, srv1947207341=53, srv2098764003=62, srv1392032420=18, srv1112843246=5, srv1645796270=37, srv263976351=67, srv953731216=99, srv1986419332=55, srv1261486521=11, srv1906124409=49, srv170679345=40, srv906622147=96, srv157693946=34, srv747350067=89, srv748961219=90, srv272044504=69, srv1562617009=31, srv1998163778=56, srv1691191096=39, srv2027580705=57, srv1419409168=20, srv1112708340=4, srv1373759038=16, srv1530540232=30, srv581316021=84} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-27 14:54:04,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-27 14:54:04,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-27 14:54:04,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-27 14:54:04,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-27 14:54:04,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-27 14:54:04,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-27 14:54:04,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-27 14:54:04,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-27 14:54:04,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-27 14:54:04,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-27 14:54:04,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-27 14:54:04,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-27 14:54:04,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-27 14:54:04,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-27 14:54:04,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-27 14:54:04,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-27 14:54:04,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-27 14:54:04,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-27 14:54:04,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-27 14:54:04,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-27 14:54:04,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-27 14:54:04,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-27 14:54:04,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-27 14:54:04,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-27 14:54:04,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-27 14:54:04,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-27 14:54:04,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-27 14:54:04,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-27 14:54:04,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-27 14:54:04,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-27 14:54:04,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-27 14:54:04,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-27 14:54:04,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-27 14:54:04,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-27 14:54:04,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-27 14:54:04,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-27 14:54:04,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-27 14:54:04,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-27 14:54:04,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-27 14:54:04,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-27 14:54:04,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-27 14:54:04,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-27 14:54:04,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-27 14:54:04,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-27 14:54:04,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-27 14:54:04,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-27 14:54:04,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-27 14:54:04,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-27 14:54:04,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-27 14:54:04,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-27 14:54:04,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-27 14:54:04,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-27 14:54:04,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-27 14:54:04,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-27 14:54:04,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-27 14:54:04,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-27 14:54:04,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-27 14:54:04,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-27 14:54:04,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-27 14:54:04,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-27 14:54:04,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-27 14:54:04,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-27 14:54:04,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-27 14:54:04,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-27 14:54:04,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-27 14:54:04,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-27 14:54:04,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-27 14:54:04,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-27 14:54:04,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-27 14:54:04,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-27 14:54:04,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-27 14:54:04,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-27 14:54:04,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-27 14:54:04,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-27 14:54:04,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-27 14:54:04,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-27 14:54:04,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-27 14:54:04,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-27 14:54:04,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-27 14:54:04,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-27 14:54:04,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-27 14:54:04,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-27 14:54:04,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-27 14:54:04,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-27 14:54:04,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-27 14:54:04,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-27 14:54:04,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-27 14:54:04,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-27 14:54:04,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-27 14:54:04,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-27 14:54:04,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-27 14:54:04,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-27 14:54:04,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-27 14:54:04,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-27 14:54:04,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-27 14:54:04,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-27 14:54:04,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-27 14:54:04,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-27 14:54:04,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-27 14:54:04,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-27 14:54:04,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-27 14:54:04,660 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-27 14:54:04,661 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-27 14:54:04,661 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-27 14:54:05,063 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 564 ms to try 5000 different iterations.  Found a solution that moves 4663 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18189217034718008. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038005847503304595, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.053486732256674586, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5181111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038005847503304595, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17758333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2395, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-27 14:54:05,087 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1153703888:82 , srv2040726867:83 , srv34418757:83 , srv550977424:83 , srv817231890:83 , srv953731216:83 , srv1054617934:84 , srv1112843246:84 , srv1261486521:84 , srv1451424236:84 , srv1567495364:84 , srv1799161503:84 , srv2042805583:84 , srv265284146:84 , srv310332301:84 , srv397746616:84 , srv512414194:84 , srv1191283213:85 , srv1373759038:85 , srv149352909:85 , srv1518696570:85 , srv1645796270:85 , srv1691191096:85 , srv1773321675:85 , srv1906124409:85 , srv1908650428:85 , srv1986419332:85 , srv322823046:85 , srv906622147:85 , srv931179430:85 , srv1000856827:86 , srv1002504854:86 , srv1146389561:86 , srv1289099110:86 , srv1410800255:86 , srv1498991057:86 , srv1516053655:86 , srv1523382489:86 , srv1593036941:86 , srv170679345:86 , srv1725925292:86 , srv1809988751:86 , srv2098764003:86 , srv263976351:86 , srv324635108:86 , srv480227054:86 , srv581683752:86 , srv747350067:86 , srv816588091:86 , srv849069256:86 , srv1112708340:87 , srv1177735838:87 , srv1355171598:87 , srv1443289231:87 , srv1454497578:87 , srv1562617009:87 , srv1778463864:87 , srv1783285850:87 , srv1935276149:87 , srv228321302:87 , srv376824824:87 , srv392855187:87 , srv512330658:87 , srv538012091:87 , srv857467757:87 , srv913276901:87 , srv1039380747:88 , srv1232275388:88 , srv129712100:88 , srv1392032420:88 , srv1419409168:88 , srv1447638751:88 , srv1606920511:88 , srv1756255529:88 , srv192044803:88 , srv1947207341:88 , srv1973618242:88 , srv2126817073:88 , srv2137980786:88 , srv236177160:88 , srv272044504:88 , srv510788721:88 , srv581316021:88 , srv683021025:88 , srv138281273:89 , srv166365277:89 , srv2027580705:89 , srv2039211357:89 , srv2057879659:89 , srv690507778:89 , srv705357198:89 , srv781321102:89 , srv748961219:90 , srv1530540232:91 , srv1563511613:91 , srv1734138194:91 , srv1998163778:91 , srv1328352381:92 , srv37651788:92 , srv157693946:428 }
2023-11-27 14:54:05,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-27 14:54:05,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv129712100=13, srv376824824=75, srv1328352381=14, srv310332301=70, srv1498991057=26, srv2126817073=63, srv849069256=94, srv1783285850=46, srv265284146=68, srv1146389561=6, srv857467757=95, srv1289099110=12, srv1935276149=52, srv1523382489=29, srv510788721=79, srv1518696570=28, srv1054617934=3, srv2137980786=64, srv1447638751=22, srv37651788=74, srv817231890=93, srv166365277=38, srv781321102=91, srv1756255529=43, srv392855187=76, srv581683752=85, srv1191283213=9, srv1778463864=45, srv1232275388=10, srv1451424236=23, srv1454497578=24, srv324635108=72, srv34418757=73, srv705357198=88, srv1443289231=21, srv2040726867=59, srv1000856827=0, srv192044803=51, srv1153703888=7, srv1799161503=47, srv480227054=78, srv149352909=25, srv1355171598=15, srv228321302=65, srv397746616=77, srv1177735838=8, srv816588091=92, srv1809988751=48, srv1039380747=2, srv236177160=66, srv1725925292=41, srv138281273=17, srv322823046=71, srv538012091=82, srv1593036941=35, srv2042805583=60, srv512414194=81, srv2039211357=58, srv1908650428=50, srv1773321675=44, srv1606920511=36, srv913276901=97, srv550977424=83, srv512330658=80, srv1973618242=54, srv683021025=86, srv1567495364=33, srv1002504854=1, srv1410800255=19, srv931179430=98, srv1516053655=27, srv690507778=87, srv1734138194=42, srv2057879659=61, srv1563511613=32, srv1947207341=53, srv2098764003=62, srv1392032420=18, srv1112843246=5, srv1645796270=37, srv263976351=67, srv953731216=99, srv1986419332=55, srv1261486521=11, srv1906124409=49, srv170679345=40, srv906622147=96, srv157693946=34, srv747350067=89, srv748961219=90, srv272044504=69, srv1562617009=31, srv1998163778=56, srv1691191096=39, srv2027580705=57, srv1419409168=20, srv1112708340=4, srv1373759038=16, srv1530540232=30, srv581316021=84} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-27 14:54:05,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-27 14:54:05,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-27 14:54:05,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-27 14:54:05,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-27 14:54:05,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-27 14:54:05,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-27 14:54:05,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-27 14:54:05,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-27 14:54:05,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-27 14:54:05,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-27 14:54:05,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-27 14:54:05,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-27 14:54:05,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-27 14:54:05,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-27 14:54:05,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-27 14:54:05,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-27 14:54:05,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-27 14:54:05,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-27 14:54:05,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-27 14:54:05,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-27 14:54:05,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-27 14:54:05,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-27 14:54:05,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-27 14:54:05,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-27 14:54:05,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-27 14:54:05,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-27 14:54:05,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-27 14:54:05,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-27 14:54:05,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-27 14:54:05,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-27 14:54:05,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-27 14:54:05,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-27 14:54:05,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-27 14:54:05,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-27 14:54:05,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-27 14:54:05,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-27 14:54:05,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-27 14:54:05,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-27 14:54:05,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-27 14:54:05,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-27 14:54:05,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-27 14:54:05,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-27 14:54:05,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-27 14:54:05,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-27 14:54:05,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-27 14:54:05,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-27 14:54:05,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-27 14:54:05,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-27 14:54:05,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-27 14:54:05,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-27 14:54:05,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-27 14:54:05,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-27 14:54:05,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-27 14:54:05,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-27 14:54:05,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-27 14:54:05,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-27 14:54:05,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-27 14:54:05,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-27 14:54:05,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-27 14:54:05,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-27 14:54:05,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-27 14:54:05,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-27 14:54:05,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-27 14:54:05,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-27 14:54:05,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-27 14:54:05,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-27 14:54:05,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-27 14:54:05,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-27 14:54:05,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-27 14:54:05,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-27 14:54:05,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-27 14:54:05,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-27 14:54:05,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-27 14:54:05,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-27 14:54:05,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-27 14:54:05,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-27 14:54:05,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-27 14:54:05,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-27 14:54:05,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-27 14:54:05,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-27 14:54:05,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-27 14:54:05,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-27 14:54:05,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-27 14:54:05,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-27 14:54:05,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-27 14:54:05,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-27 14:54:05,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-27 14:54:05,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-27 14:54:05,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-27 14:54:05,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-27 14:54:05,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-27 14:54:05,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-27 14:54:05,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-27 14:54:05,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-27 14:54:05,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-27 14:54:05,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-27 14:54:05,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-27 14:54:05,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-27 14:54:05,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-27 14:54:05,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-27 14:54:05,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-27 14:54:05,265 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038005847503304595, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.053486732256674586, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038005847503304595, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17758333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2395, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-27 14:54:05,265 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-27 14:54:05,265 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18185951932331573, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038005847503304595, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.053486732256674586, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038005847503304595, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17758333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2395, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-27 14:54:05,547 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 400 ms to try 5000 different iterations.  Found a solution that moves 2570 regions; Going from a computed imbalance of 0.18185951932331573 to a new imbalance of 0.015267526794419528. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022499393792667743); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03700335168487718, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.28555555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022499393792667743); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012333333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-27 14:54:05,560 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1328352381:79 , srv37651788:80 , srv138281273:81 , srv1563511613:81 , srv166365277:81 , srv690507778:81 , srv781321102:81 , srv1998163778:82 , srv748961219:82 , srv1039380747:83 , srv1355171598:83 , srv1530540232:83 , srv1734138194:83 , srv1935276149:83 , srv1973618242:83 , srv2057879659:83 , srv2137980786:83 , srv236177160:83 , srv129712100:84 , srv1419409168:84 , srv1454497578:84 , srv1562617009:84 , srv1606920511:84 , srv1778463864:84 , srv1783285850:84 , srv2039211357:84 , srv2126817073:84 , srv272044504:84 , srv324635108:84 , srv392855187:84 , srv512330658:84 , srv705357198:84 , srv849069256:84 , srv857467757:84 , srv1112708340:85 , srv1191283213:85 , srv1392032420:85 , srv1410800255:85 , srv1447638751:85 , srv1523382489:85 , srv1593036941:85 , srv1691191096:85 , srv170679345:85 , srv1756255529:85 , srv1773321675:85 , srv1809988751:85 , srv192044803:85 , srv2027580705:85 , srv2098764003:85 , srv322823046:85 , srv376824824:85 , srv510788721:85 , srv747350067:85 , srv816588091:85 , srv1000856827:86 , srv1002504854:86 , srv1112843246:86 , srv1177735838:86 , srv1498991057:86 , srv1516053655:86 , srv1518696570:86 , srv1725925292:86 , srv1908650428:86 , srv1947207341:86 , srv228321302:86 , srv263976351:86 , srv581316021:86 , srv683021025:86 , srv1054617934:87 , srv1232275388:87 , srv1289099110:87 , srv1443289231:87 , srv149352909:87 , srv265284146:87 , srv480227054:87 , srv538012091:87 , srv581683752:87 , srv906622147:87 , srv913276901:87 , srv931179430:87 , srv1153703888:88 , srv1261486521:88 , srv1373759038:88 , srv1451424236:88 , srv1645796270:88 , srv2040726867:88 , srv2042805583:88 , srv310332301:88 , srv550977424:88 , srv953731216:88 , srv1146389561:89 , srv1567495364:89 , srv1799161503:89 , srv512414194:89 , srv817231890:89 , srv1906124409:90 , srv34418757:90 , srv1986419332:91 , srv397746616:91 , srv157693946:546 }
2023-11-27 14:54:05,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-27 14:54:05,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv129712100=13, srv376824824=75, srv1328352381=14, srv310332301=70, srv1498991057=26, srv2126817073=63, srv849069256=94, srv1783285850=46, srv265284146=68, srv1146389561=6, srv857467757=95, srv1289099110=12, srv1935276149=52, srv1523382489=29, srv510788721=79, srv1518696570=28, srv1054617934=3, srv2137980786=64, srv1447638751=22, srv37651788=74, srv817231890=93, srv166365277=38, srv781321102=91, srv1756255529=43, srv392855187=76, srv581683752=85, srv1191283213=9, srv1778463864=45, srv1232275388=10, srv1451424236=23, srv1454497578=24, srv324635108=72, srv34418757=73, srv705357198=88, srv1443289231=21, srv2040726867=59, srv1000856827=0, srv192044803=51, srv1153703888=7, srv1799161503=47, srv480227054=78, srv149352909=25, srv1355171598=15, srv228321302=65, srv397746616=77, srv1177735838=8, srv816588091=92, srv1809988751=48, srv1039380747=2, srv236177160=66, srv1725925292=41, srv138281273=17, srv322823046=71, srv538012091=82, srv1593036941=35, srv2042805583=60, srv512414194=81, srv2039211357=58, srv1908650428=50, srv1773321675=44, srv1606920511=36, srv913276901=97, srv550977424=83, srv512330658=80, srv1973618242=54, srv683021025=86, srv1567495364=33, srv1002504854=1, srv1410800255=19, srv931179430=98, srv1516053655=27, srv690507778=87, srv1734138194=42, srv2057879659=61, srv1563511613=32, srv1947207341=53, srv2098764003=62, srv1392032420=18, srv1112843246=5, srv1645796270=37, srv263976351=67, srv953731216=99, srv1986419332=55, srv1261486521=11, srv1906124409=49, srv170679345=40, srv906622147=96, srv157693946=34, srv747350067=89, srv748961219=90, srv272044504=69, srv1562617009=31, srv1998163778=56, srv1691191096=39, srv2027580705=57, srv1419409168=20, srv1112708340=4, srv1373759038=16, srv1530540232=30, srv581316021=84} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-27 14:54:05,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-27 14:54:05,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-27 14:54:05,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-27 14:54:05,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-27 14:54:05,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-27 14:54:05,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-27 14:54:05,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-27 14:54:05,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-27 14:54:05,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-27 14:54:05,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-27 14:54:05,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-27 14:54:05,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-27 14:54:05,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-27 14:54:05,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-27 14:54:05,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-27 14:54:05,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-27 14:54:05,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-27 14:54:05,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-27 14:54:05,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-27 14:54:05,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-27 14:54:05,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-27 14:54:05,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-27 14:54:05,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-27 14:54:05,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-27 14:54:05,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-27 14:54:05,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-27 14:54:05,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-27 14:54:05,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-27 14:54:05,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-27 14:54:05,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-27 14:54:05,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-27 14:54:05,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-27 14:54:05,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-27 14:54:05,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-27 14:54:05,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-27 14:54:05,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-27 14:54:05,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-27 14:54:05,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-27 14:54:05,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-27 14:54:05,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-27 14:54:05,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-27 14:54:05,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-27 14:54:05,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-27 14:54:05,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-27 14:54:05,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-27 14:54:05,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-27 14:54:05,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-27 14:54:05,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-27 14:54:05,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-27 14:54:05,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-27 14:54:05,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-27 14:54:05,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-27 14:54:05,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-27 14:54:05,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-27 14:54:05,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-27 14:54:05,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-27 14:54:05,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-27 14:54:05,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-27 14:54:05,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-27 14:54:05,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-27 14:54:05,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-27 14:54:05,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-27 14:54:05,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-27 14:54:05,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-27 14:54:05,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-27 14:54:05,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-27 14:54:05,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-27 14:54:05,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-27 14:54:05,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-27 14:54:05,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-27 14:54:05,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-27 14:54:05,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-27 14:54:05,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-27 14:54:05,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-27 14:54:05,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-27 14:54:05,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-27 14:54:05,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-27 14:54:05,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-27 14:54:05,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-27 14:54:05,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-27 14:54:05,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-27 14:54:05,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-27 14:54:05,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-27 14:54:05,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-27 14:54:05,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-27 14:54:05,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-27 14:54:05,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-27 14:54:05,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-27 14:54:05,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-27 14:54:05,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-27 14:54:05,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-27 14:54:05,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-27 14:54:05,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-27 14:54:05,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-27 14:54:05,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-27 14:54:05,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-27 14:54:05,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-27 14:54:05,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-27 14:54:05,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-27 14:54:05,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-27 14:54:05,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-27 14:54:05,784 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022499393792667743); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03700335168487718, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022499393792667743); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012333333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-27 14:54:05,784 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-27 14:54:05,784 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015249531269793469, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022499393792667743); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03700335168487718, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022499393792667743); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012333333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-27 14:54:06,153 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 530 ms to try 5000 different iterations.  Found a solution that moves 967 regions; Going from a computed imbalance of 0.015249531269793469 to a new imbalance of 6.417901072861923E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005322847904476634); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00704323668071124); MoveCostFunction : (multiplier=7.0, imbalance=0.10744444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005322847904476634); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-27 14:54:06,155 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1146389561:82 , srv1986419332:82 , srv1232275388:83 , srv1530540232:83 , srv397746616:83 , srv538012091:83 , srv683021025:83 , srv1392032420:84 , srv1443289231:84 , srv1447638751:84 , srv1516053655:84 , srv1734138194:84 , srv1799161503:84 , srv1906124409:84 , srv192044803:84 , srv2027580705:84 , srv228321302:84 , srv581316021:84 , srv705357198:84 , srv748961219:84 , srv913276901:84 , srv1000856827:85 , srv1177735838:85 , srv1289099110:85 , srv129712100:85 , srv1373759038:85 , srv1451424236:85 , srv1498991057:85 , srv1563511613:85 , srv1567495364:85 , srv1606920511:85 , srv1645796270:85 , srv1725925292:85 , srv1756255529:85 , srv1947207341:85 , srv1998163778:85 , srv2039211357:85 , srv2126817073:85 , srv272044504:85 , srv310332301:85 , srv480227054:85 , srv510788721:85 , srv512414194:85 , srv581683752:85 , srv1002504854:86 , srv1039380747:86 , srv1054617934:86 , srv1112708340:86 , srv1261486521:86 , srv1328352381:86 , srv1419409168:86 , srv149352909:86 , srv1518696570:86 , srv1523382489:86 , srv1562617009:86 , srv1593036941:86 , srv166365277:86 , srv1783285850:86 , srv1908650428:86 , srv1973618242:86 , srv2042805583:86 , srv2057879659:86 , srv2098764003:86 , srv2137980786:86 , srv263976351:86 , srv322823046:86 , srv34418757:86 , srv37651788:86 , srv376824824:86 , srv512330658:86 , srv550977424:86 , srv816588091:86 , srv817231890:86 , srv857467757:86 , srv906622147:86 , srv931179430:86 , srv953731216:86 , srv1153703888:87 , srv1191283213:87 , srv1355171598:87 , srv138281273:87 , srv1410800255:87 , srv1454497578:87 , srv1691191096:87 , srv170679345:87 , srv1809988751:87 , srv2040726867:87 , srv236177160:87 , srv265284146:87 , srv392855187:87 , srv747350067:87 , srv781321102:87 , srv1112843246:88 , srv1773321675:88 , srv1778463864:88 , srv1935276149:88 , srv324635108:88 , srv690507778:88 , srv849069256:88 , srv157693946:531 }
2023-11-27 14:54:06,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-27 14:54:06,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv129712100=13, srv376824824=75, srv1328352381=14, srv310332301=70, srv1498991057=26, srv2126817073=63, srv849069256=94, srv1783285850=46, srv265284146=68, srv1146389561=6, srv857467757=95, srv1289099110=12, srv1935276149=52, srv1523382489=29, srv510788721=79, srv1518696570=28, srv1054617934=3, srv2137980786=64, srv1447638751=22, srv37651788=74, srv817231890=93, srv166365277=38, srv781321102=91, srv1756255529=43, srv392855187=76, srv581683752=85, srv1191283213=9, srv1778463864=45, srv1232275388=10, srv1451424236=23, srv1454497578=24, srv324635108=72, srv34418757=73, srv705357198=88, srv1443289231=21, srv2040726867=59, srv1000856827=0, srv192044803=51, srv1153703888=7, srv1799161503=47, srv480227054=78, srv149352909=25, srv1355171598=15, srv228321302=65, srv397746616=77, srv1177735838=8, srv816588091=92, srv1809988751=48, srv1039380747=2, srv236177160=66, srv1725925292=41, srv138281273=17, srv322823046=71, srv538012091=82, srv1593036941=35, srv2042805583=60, srv512414194=81, srv2039211357=58, srv1908650428=50, srv1773321675=44, srv1606920511=36, srv913276901=97, srv550977424=83, srv512330658=80, srv1973618242=54, srv683021025=86, srv1567495364=33, srv1002504854=1, srv1410800255=19, srv931179430=98, srv1516053655=27, srv690507778=87, srv1734138194=42, srv2057879659=61, srv1563511613=32, srv1947207341=53, srv2098764003=62, srv1392032420=18, srv1112843246=5, srv1645796270=37, srv263976351=67, srv953731216=99, srv1986419332=55, srv1261486521=11, srv1906124409=49, srv170679345=40, srv906622147=96, srv157693946=34, srv747350067=89, srv748961219=90, srv272044504=69, srv1562617009=31, srv1998163778=56, srv1691191096=39, srv2027580705=57, srv1419409168=20, srv1112708340=4, srv1373759038=16, srv1530540232=30, srv581316021=84} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-27 14:54:06,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-27 14:54:06,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-27 14:54:06,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-27 14:54:06,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-27 14:54:06,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-27 14:54:06,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-27 14:54:06,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-27 14:54:06,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-27 14:54:06,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-27 14:54:06,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-27 14:54:06,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-27 14:54:06,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-27 14:54:06,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-27 14:54:06,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-27 14:54:06,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-27 14:54:06,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-27 14:54:06,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-27 14:54:06,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-27 14:54:06,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-27 14:54:06,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-27 14:54:06,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-27 14:54:06,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-27 14:54:06,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-27 14:54:06,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-27 14:54:06,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-27 14:54:06,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-27 14:54:06,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-27 14:54:06,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-27 14:54:06,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-27 14:54:06,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-27 14:54:06,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-27 14:54:06,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-27 14:54:06,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-27 14:54:06,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-27 14:54:06,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-27 14:54:06,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-27 14:54:06,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-27 14:54:06,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-27 14:54:06,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-27 14:54:06,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-27 14:54:06,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-27 14:54:06,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-27 14:54:06,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-27 14:54:06,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-27 14:54:06,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-27 14:54:06,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-27 14:54:06,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-27 14:54:06,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-27 14:54:06,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-27 14:54:06,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-27 14:54:06,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-27 14:54:06,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-27 14:54:06,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-27 14:54:06,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-27 14:54:06,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-27 14:54:06,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-27 14:54:06,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-27 14:54:06,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-27 14:54:06,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-27 14:54:06,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-27 14:54:06,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-27 14:54:06,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-27 14:54:06,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-27 14:54:06,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-27 14:54:06,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-27 14:54:06,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-27 14:54:06,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-27 14:54:06,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-27 14:54:06,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-27 14:54:06,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-27 14:54:06,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-27 14:54:06,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-27 14:54:06,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-27 14:54:06,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-27 14:54:06,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-27 14:54:06,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-27 14:54:06,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-27 14:54:06,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-27 14:54:06,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-27 14:54:06,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-27 14:54:06,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-27 14:54:06,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-27 14:54:06,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-27 14:54:06,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-27 14:54:06,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-27 14:54:06,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-27 14:54:06,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-27 14:54:06,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-27 14:54:06,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-27 14:54:06,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-27 14:54:06,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-27 14:54:06,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-27 14:54:06,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-27 14:54:06,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-27 14:54:06,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-27 14:54:06,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-27 14:54:06,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-27 14:54:06,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-27 14:54:06,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-27 14:54:06,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-27 14:54:06,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-27 14:54:06,312 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005322847904476634); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00704323668071124); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005322847904476634); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-27 14:54:06,312 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-27 14:54:06,312 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.35019028565025E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005322847904476634); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00704323668071124); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005322847904476634); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-27 14:54:06,703 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 498 ms to try 5000 different iterations.  Found a solution that moves 234 regions; Going from a computed imbalance of 6.35019028565025E-4 to a new imbalance of 4.163989513349733E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.026, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-27 14:54:06,704 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1778463864:83 , srv1947207341:83 , srv1002504854:84 , srv1054617934:84 , srv1112708340:84 , srv1112843246:84 , srv1146389561:84 , srv1177735838:84 , srv1232275388:84 , srv1261486521:84 , srv1289099110:84 , srv1328352381:84 , srv1355171598:84 , srv1373759038:84 , srv1392032420:84 , srv1410800255:84 , srv1419409168:84 , srv1454497578:84 , srv149352909:84 , srv1562617009:84 , srv1645796270:84 , srv170679345:84 , srv1734138194:84 , srv1756255529:84 , srv1809988751:84 , srv1906124409:84 , srv1986419332:84 , srv1998163778:84 , srv2027580705:84 , srv2039211357:84 , srv2040726867:84 , srv2042805583:84 , srv2057879659:84 , srv236177160:84 , srv263976351:84 , srv265284146:84 , srv34418757:84 , srv37651788:84 , srv376824824:84 , srv392855187:84 , srv397746616:84 , srv480227054:84 , srv510788721:84 , srv512414194:84 , srv581316021:84 , srv581683752:84 , srv690507778:84 , srv747350067:84 , srv817231890:84 , srv849069256:84 , srv906622147:84 , srv913276901:84 , srv931179430:84 , srv1000856827:85 , srv1039380747:85 , srv1153703888:85 , srv1191283213:85 , srv129712100:85 , srv138281273:85 , srv1443289231:85 , srv1447638751:85 , srv1451424236:85 , srv1498991057:85 , srv1518696570:85 , srv1523382489:85 , srv1530540232:85 , srv1563511613:85 , srv1567495364:85 , srv1593036941:85 , srv1606920511:85 , srv1691191096:85 , srv1725925292:85 , srv1773321675:85 , srv1783285850:85 , srv1799161503:85 , srv1908650428:85 , srv192044803:85 , srv1935276149:85 , srv1973618242:85 , srv2098764003:85 , srv2126817073:85 , srv2137980786:85 , srv228321302:85 , srv272044504:85 , srv310332301:85 , srv324635108:85 , srv512330658:85 , srv538012091:85 , srv550977424:85 , srv683021025:85 , srv705357198:85 , srv748961219:85 , srv781321102:85 , srv816588091:85 , srv857467757:85 , srv953731216:85 , srv1516053655:86 , srv166365277:86 , srv322823046:86 , srv157693946:637 }
2023-11-27 14:54:06,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-27 14:54:06,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv129712100=13, srv376824824=75, srv1328352381=14, srv310332301=70, srv1498991057=26, srv2126817073=63, srv849069256=94, srv1783285850=46, srv265284146=68, srv1146389561=6, srv857467757=95, srv1289099110=12, srv1935276149=52, srv1523382489=29, srv510788721=79, srv1518696570=28, srv1054617934=3, srv2137980786=64, srv1447638751=22, srv37651788=74, srv817231890=93, srv166365277=38, srv781321102=91, srv1756255529=43, srv392855187=76, srv581683752=85, srv1191283213=9, srv1778463864=45, srv1232275388=10, srv1451424236=23, srv1454497578=24, srv324635108=72, srv34418757=73, srv705357198=88, srv1443289231=21, srv2040726867=59, srv1000856827=0, srv192044803=51, srv1153703888=7, srv1799161503=47, srv480227054=78, srv149352909=25, srv1355171598=15, srv228321302=65, srv397746616=77, srv1177735838=8, srv816588091=92, srv1809988751=48, srv1039380747=2, srv236177160=66, srv1725925292=41, srv138281273=17, srv322823046=71, srv538012091=82, srv1593036941=35, srv2042805583=60, srv512414194=81, srv2039211357=58, srv1908650428=50, srv1773321675=44, srv1606920511=36, srv913276901=97, srv550977424=83, srv512330658=80, srv1973618242=54, srv683021025=86, srv1567495364=33, srv1002504854=1, srv1410800255=19, srv931179430=98, srv1516053655=27, srv690507778=87, srv1734138194=42, srv2057879659=61, srv1563511613=32, srv1947207341=53, srv2098764003=62, srv1392032420=18, srv1112843246=5, srv1645796270=37, srv263976351=67, srv953731216=99, srv1986419332=55, srv1261486521=11, srv1906124409=49, srv170679345=40, srv906622147=96, srv157693946=34, srv747350067=89, srv748961219=90, srv272044504=69, srv1562617009=31, srv1998163778=56, srv1691191096=39, srv2027580705=57, srv1419409168=20, srv1112708340=4, srv1373759038=16, srv1530540232=30, srv581316021=84} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-27 14:54:06,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-27 14:54:06,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-27 14:54:06,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-27 14:54:06,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-27 14:54:06,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-27 14:54:06,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-27 14:54:06,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-27 14:54:06,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-27 14:54:06,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-27 14:54:06,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-27 14:54:06,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-27 14:54:06,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-27 14:54:06,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-27 14:54:06,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-27 14:54:06,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-27 14:54:06,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-27 14:54:06,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-27 14:54:06,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-27 14:54:06,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-27 14:54:06,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-27 14:54:06,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-27 14:54:06,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-27 14:54:06,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-27 14:54:06,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-27 14:54:06,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-27 14:54:06,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-27 14:54:06,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-27 14:54:06,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-27 14:54:06,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-27 14:54:06,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-27 14:54:06,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-27 14:54:06,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-27 14:54:06,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-27 14:54:06,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-27 14:54:06,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-27 14:54:06,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-27 14:54:06,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-27 14:54:06,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-27 14:54:06,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-27 14:54:06,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-27 14:54:06,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-27 14:54:06,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-27 14:54:06,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-27 14:54:06,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-27 14:54:06,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-27 14:54:06,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-27 14:54:06,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-27 14:54:06,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-27 14:54:06,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-27 14:54:06,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-27 14:54:06,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-27 14:54:06,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-27 14:54:06,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-27 14:54:06,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-27 14:54:06,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-27 14:54:06,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-27 14:54:06,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-27 14:54:06,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-27 14:54:06,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-27 14:54:06,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-27 14:54:06,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-27 14:54:06,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-27 14:54:06,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-27 14:54:06,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-27 14:54:06,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-27 14:54:06,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-27 14:54:06,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-27 14:54:06,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-27 14:54:06,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-27 14:54:06,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-27 14:54:06,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-27 14:54:06,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-27 14:54:06,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-27 14:54:06,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-27 14:54:06,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-27 14:54:06,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-27 14:54:06,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-27 14:54:06,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-27 14:54:06,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-27 14:54:06,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-27 14:54:06,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-27 14:54:06,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-27 14:54:06,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-27 14:54:06,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-27 14:54:06,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-27 14:54:06,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-27 14:54:06,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-27 14:54:06,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-27 14:54:06,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-27 14:54:06,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-27 14:54:06,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-27 14:54:06,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-27 14:54:06,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-27 14:54:06,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-27 14:54:06,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-27 14:54:06,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-27 14:54:06,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-27 14:54:06,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-27 14:54:06,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-27 14:54:06,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-27 14:54:06,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-27 14:54:06,843 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-27 14:54:06,843 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-27 14:54:06,843 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.000139211307007E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-27 14:54:07,191 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 456 ms to try 5000 different iterations.  Found a solution that moves 17 regions; Going from a computed imbalance of 4.000139211307007E-5 to a new imbalance of 7.808458344569857E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.001888888888888889); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-27 14:54:07,192 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1443289231:83 , srv1567495364:83 , srv1773321675:83 , srv1906124409:83 , srv1935276149:83 , srv324635108:83 , srv34418757:83 , srv1000856827:84 , srv1002504854:84 , srv1039380747:84 , srv1112708340:84 , srv1112843246:84 , srv1153703888:84 , srv1177735838:84 , srv1191283213:84 , srv1232275388:84 , srv1261486521:84 , srv1289099110:84 , srv129712100:84 , srv1373759038:84 , srv138281273:84 , srv1410800255:84 , srv1419409168:84 , srv1447638751:84 , srv1451424236:84 , srv1454497578:84 , srv149352909:84 , srv1498991057:84 , srv1516053655:84 , srv1518696570:84 , srv1523382489:84 , srv1530540232:84 , srv1563511613:84 , srv1593036941:84 , srv1606920511:84 , srv1645796270:84 , srv166365277:84 , srv1691191096:84 , srv170679345:84 , srv1725925292:84 , srv1734138194:84 , srv1756255529:84 , srv1778463864:84 , srv1783285850:84 , srv1799161503:84 , srv1809988751:84 , srv1908650428:84 , srv192044803:84 , srv1947207341:84 , srv1973618242:84 , srv1986419332:84 , srv1998163778:84 , srv2027580705:84 , srv2039211357:84 , srv2040726867:84 , srv2042805583:84 , srv2057879659:84 , srv2098764003:84 , srv2126817073:84 , srv2137980786:84 , srv228321302:84 , srv236177160:84 , srv263976351:84 , srv265284146:84 , srv272044504:84 , srv310332301:84 , srv322823046:84 , srv37651788:84 , srv376824824:84 , srv392855187:84 , srv397746616:84 , srv480227054:84 , srv510788721:84 , srv512330658:84 , srv512414194:84 , srv538012091:84 , srv550977424:84 , srv581316021:84 , srv581683752:84 , srv683021025:84 , srv690507778:84 , srv705357198:84 , srv747350067:84 , srv781321102:84 , srv816588091:84 , srv817231890:84 , srv849069256:84 , srv857467757:84 , srv906622147:84 , srv913276901:84 , srv931179430:84 , srv953731216:84 , srv1054617934:85 , srv1146389561:85 , srv1328352381:85 , srv1355171598:85 , srv1392032420:85 , srv1562617009:85 , srv748961219:85 , srv157693946:684 }
2023-11-27 14:54:07,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-27 14:54:07,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv129712100=13, srv376824824=75, srv1328352381=14, srv310332301=70, srv1498991057=26, srv2126817073=63, srv849069256=94, srv1783285850=46, srv265284146=68, srv1146389561=6, srv857467757=95, srv1289099110=12, srv1935276149=52, srv1523382489=29, srv510788721=79, srv1518696570=28, srv1054617934=3, srv2137980786=64, srv1447638751=22, srv37651788=74, srv817231890=93, srv166365277=38, srv781321102=91, srv1756255529=43, srv392855187=76, srv581683752=85, srv1191283213=9, srv1778463864=45, srv1232275388=10, srv1451424236=23, srv1454497578=24, srv324635108=72, srv34418757=73, srv705357198=88, srv1443289231=21, srv2040726867=59, srv1000856827=0, srv192044803=51, srv1153703888=7, srv1799161503=47, srv480227054=78, srv149352909=25, srv1355171598=15, srv228321302=65, srv397746616=77, srv1177735838=8, srv816588091=92, srv1809988751=48, srv1039380747=2, srv236177160=66, srv1725925292=41, srv138281273=17, srv322823046=71, srv538012091=82, srv1593036941=35, srv2042805583=60, srv512414194=81, srv2039211357=58, srv1908650428=50, srv1773321675=44, srv1606920511=36, srv913276901=97, srv550977424=83, srv512330658=80, srv1973618242=54, srv683021025=86, srv1567495364=33, srv1002504854=1, srv1410800255=19, srv931179430=98, srv1516053655=27, srv690507778=87, srv1734138194=42, srv2057879659=61, srv1563511613=32, srv1947207341=53, srv2098764003=62, srv1392032420=18, srv1112843246=5, srv1645796270=37, srv263976351=67, srv953731216=99, srv1986419332=55, srv1261486521=11, srv1906124409=49, srv170679345=40, srv906622147=96, srv157693946=34, srv747350067=89, srv748961219=90, srv272044504=69, srv1562617009=31, srv1998163778=56, srv1691191096=39, srv2027580705=57, srv1419409168=20, srv1112708340=4, srv1373759038=16, srv1530540232=30, srv581316021=84} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-27 14:54:07,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-27 14:54:07,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-27 14:54:07,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-27 14:54:07,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-27 14:54:07,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-27 14:54:07,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-27 14:54:07,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-27 14:54:07,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-27 14:54:07,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-27 14:54:07,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-27 14:54:07,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-27 14:54:07,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-27 14:54:07,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-27 14:54:07,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-27 14:54:07,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-27 14:54:07,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-27 14:54:07,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-27 14:54:07,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-27 14:54:07,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-27 14:54:07,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-27 14:54:07,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-27 14:54:07,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-27 14:54:07,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-27 14:54:07,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-27 14:54:07,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-27 14:54:07,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-27 14:54:07,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-27 14:54:07,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-27 14:54:07,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-27 14:54:07,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-27 14:54:07,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-27 14:54:07,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-27 14:54:07,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-27 14:54:07,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-27 14:54:07,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-27 14:54:07,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-27 14:54:07,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-27 14:54:07,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-27 14:54:07,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-27 14:54:07,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-27 14:54:07,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-27 14:54:07,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-27 14:54:07,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-27 14:54:07,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-27 14:54:07,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-27 14:54:07,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-27 14:54:07,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-27 14:54:07,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-27 14:54:07,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-27 14:54:07,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-27 14:54:07,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-27 14:54:07,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-27 14:54:07,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-27 14:54:07,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-27 14:54:07,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-27 14:54:07,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-27 14:54:07,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-27 14:54:07,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-27 14:54:07,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-27 14:54:07,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-27 14:54:07,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-27 14:54:07,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-27 14:54:07,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-27 14:54:07,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-27 14:54:07,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-27 14:54:07,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-27 14:54:07,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-27 14:54:07,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-27 14:54:07,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-27 14:54:07,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-27 14:54:07,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-27 14:54:07,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-27 14:54:07,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-27 14:54:07,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-27 14:54:07,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-27 14:54:07,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-27 14:54:07,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-27 14:54:07,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-27 14:54:07,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-27 14:54:07,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-27 14:54:07,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-27 14:54:07,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-27 14:54:07,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-27 14:54:07,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-27 14:54:07,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-27 14:54:07,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-27 14:54:07,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-27 14:54:07,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-27 14:54:07,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-27 14:54:07,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-27 14:54:07,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-27 14:54:07,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-27 14:54:07,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-27 14:54:07,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-27 14:54:07,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-27 14:54:07,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-27 14:54:07,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-27 14:54:07,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-27 14:54:07,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-27 14:54:07,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-27 14:54:07,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-27 14:54:07,342 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.689421800350782E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-27 14:54:07,342 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1443289231:83 , srv1567495364:83 , srv1773321675:83 , srv1906124409:83 , srv1935276149:83 , srv324635108:83 , srv34418757:83 , srv1000856827:84 , srv1002504854:84 , srv1039380747:84 , srv1112708340:84 , srv1112843246:84 , srv1153703888:84 , srv1177735838:84 , srv1191283213:84 , srv1232275388:84 , srv1261486521:84 , srv1289099110:84 , srv129712100:84 , srv1373759038:84 , srv138281273:84 , srv1410800255:84 , srv1419409168:84 , srv1447638751:84 , srv1451424236:84 , srv1454497578:84 , srv149352909:84 , srv1498991057:84 , srv1516053655:84 , srv1518696570:84 , srv1523382489:84 , srv1530540232:84 , srv1563511613:84 , srv1593036941:84 , srv1606920511:84 , srv1645796270:84 , srv166365277:84 , srv1691191096:84 , srv170679345:84 , srv1725925292:84 , srv1734138194:84 , srv1756255529:84 , srv1778463864:84 , srv1783285850:84 , srv1799161503:84 , srv1809988751:84 , srv1908650428:84 , srv192044803:84 , srv1947207341:84 , srv1973618242:84 , srv1986419332:84 , srv1998163778:84 , srv2027580705:84 , srv2039211357:84 , srv2040726867:84 , srv2042805583:84 , srv2057879659:84 , srv2098764003:84 , srv2126817073:84 , srv2137980786:84 , srv228321302:84 , srv236177160:84 , srv263976351:84 , srv265284146:84 , srv272044504:84 , srv310332301:84 , srv322823046:84 , srv37651788:84 , srv376824824:84 , srv392855187:84 , srv397746616:84 , srv480227054:84 , srv510788721:84 , srv512330658:84 , srv512414194:84 , srv538012091:84 , srv550977424:84 , srv581316021:84 , srv581683752:84 , srv683021025:84 , srv690507778:84 , srv705357198:84 , srv747350067:84 , srv781321102:84 , srv816588091:84 , srv817231890:84 , srv849069256:84 , srv857467757:84 , srv906622147:84 , srv913276901:84 , srv931179430:84 , srv953731216:84 , srv1054617934:85 , srv1146389561:85 , srv1328352381:85 , srv1355171598:85 , srv1392032420:85 , srv1562617009:85 , srv748961219:85 , srv157693946:684 }
2023-11-27 14:54:07,369 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=332 (was 300) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=8809 (was 9635)
2023-11-27 14:54:07,377 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=332, ProcessCount=170, AvailableMemoryMB=8809
2023-11-27 14:54:07,378 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-27 14:54:07,378 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-11-27 14:54:07,390 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv10530318:84 , srv1113867569:84 , srv1159286023:84 , srv1277313784:84 , srv1325461109:84 , srv1335500394:84 , srv1445202253:84 , srv1472020448:84 , srv1496145116:84 , srv15010982:84 , srv1511506511:84 , srv1602281086:84 , srv1642712587:84 , srv1699196240:84 , srv1702779006:84 , srv1753028702:84 , srv1817305732:84 , srv2104641648:84 , srv291513059:84 , srv312847519:84 , srv355749901:84 , srv419653968:84 , srv535847461:84 , srv557357490:84 , srv569718862:84 , srv605989744:84 , srv700644965:84 , srv717385823:84 , srv913615718:84 , srv347129976:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-11-27 14:54:07,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-27 14:54:07,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1511506511=10, srv1277313784=3, srv355749901=21, srv1325461109=4, srv1496145116=8, srv419653968=22, srv312847519=19, srv10530318=0, srv2104641648=17, srv1702779006=14, srv1602281086=11, srv1642712587=12, srv1159286023=2, srv1472020448=7, srv700644965=27, srv1753028702=15, srv291513059=18, srv557357490=24, srv1817305732=16, srv535847461=23, srv569718862=25, srv605989744=26, srv913615718=29, srv15010982=9, srv1335500394=5, srv347129976=20, srv717385823=28, srv1113867569=1, srv1445202253=6, srv1699196240=13} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-27 14:54:07,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-27 14:54:07,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-27 14:54:07,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-27 14:54:07,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-27 14:54:07,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-27 14:54:07,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-27 14:54:07,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-27 14:54:07,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-27 14:54:07,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-27 14:54:07,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-27 14:54:07,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-27 14:54:07,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-27 14:54:07,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-27 14:54:07,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-27 14:54:07,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-27 14:54:07,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-27 14:54:07,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-27 14:54:07,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-27 14:54:07,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-27 14:54:07,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-27 14:54:07,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-27 14:54:07,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-27 14:54:07,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-27 14:54:07,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-27 14:54:07,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-27 14:54:07,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-27 14:54:07,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-27 14:54:07,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-27 14:54:07,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-27 14:54:07,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-27 14:54:07,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-11-27 14:54:07,433 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0936951461952875, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-27 14:54:07,434 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912488273610514, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0936951461952875, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-11-27 14:56:07,411 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5453950 different iterations.  Found a solution that moves 1852 regions; Going from a computed imbalance of 0.9912488273610514 to a new imbalance of 4.322660390073086E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6859259259259259, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-27 14:56:07,417 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv10530318:90 , srv1113867569:90 , srv1159286023:90 , srv1277313784:90 , srv1325461109:90 , srv1335500394:90 , srv1445202253:90 , srv1472020448:90 , srv1496145116:90 , srv15010982:90 , srv1511506511:90 , srv1602281086:90 , srv1642712587:90 , srv1699196240:90 , srv1702779006:90 , srv1753028702:90 , srv1817305732:90 , srv2104641648:90 , srv291513059:90 , srv312847519:90 , srv347129976:90 , srv355749901:90 , srv419653968:90 , srv535847461:90 , srv557357490:90 , srv569718862:90 , srv605989744:90 , srv700644965:90 , srv717385823:90 , srv913615718:90 }
2023-11-27 14:56:07,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-27 14:56:07,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1511506511=10, srv1277313784=3, srv355749901=21, srv1325461109=4, srv1496145116=8, srv419653968=22, srv312847519=19, srv10530318=0, srv2104641648=17, srv1702779006=14, srv1602281086=11, srv1642712587=12, srv1159286023=2, srv1472020448=7, srv700644965=27, srv1753028702=15, srv291513059=18, srv557357490=24, srv1817305732=16, srv535847461=23, srv569718862=25, srv605989744=26, srv913615718=29, srv15010982=9, srv1335500394=5, srv347129976=20, srv717385823=28, srv1113867569=1, srv1445202253=6, srv1699196240=13} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-27 14:56:07,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-27 14:56:07,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-27 14:56:07,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-27 14:56:07,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-27 14:56:07,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-27 14:56:07,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-27 14:56:07,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-27 14:56:07,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-27 14:56:07,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-27 14:56:07,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-27 14:56:07,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-27 14:56:07,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-27 14:56:07,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-27 14:56:07,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-27 14:56:07,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-27 14:56:07,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-27 14:56:07,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-27 14:56:07,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-27 14:56:07,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-27 14:56:07,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-27 14:56:07,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-27 14:56:07,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-27 14:56:07,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-27 14:56:07,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-27 14:56:07,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-27 14:56:07,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-27 14:56:07,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-27 14:56:07,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-27 14:56:07,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-27 14:56:07,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-27 14:56:07,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-11-27 14:56:07,435 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-27 14:56:07,435 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv10530318:90 , srv1113867569:90 , srv1159286023:90 , srv1277313784:90 , srv1325461109:90 , srv1335500394:90 , srv1445202253:90 , srv1472020448:90 , srv1496145116:90 , srv15010982:90 , srv1511506511:90 , srv1602281086:90 , srv1642712587:90 , srv1699196240:90 , srv1702779006:90 , srv1753028702:90 , srv1817305732:90 , srv2104641648:90 , srv291513059:90 , srv312847519:90 , srv347129976:90 , srv355749901:90 , srv419653968:90 , srv535847461:90 , srv557357490:90 , srv569718862:90 , srv605989744:90 , srv700644965:90 , srv717385823:90 , srv913615718:90 }
2023-11-27 14:56:07,447 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=182 (was 332), ProcessCount=168 (was 170), AvailableMemoryMB=8114 (was 8809)