2023-06-07 02:00:26,030 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-06-07 02:00:26,259 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-06-07 02:00:26,353 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-06-07 02:00:26,406 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-06-07 02:00:26,411 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-06-07 02:00:26,422 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=134, ProcessCount=169, AvailableMemoryMB=5658
2023-06-07 02:00:26,424 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-06-07 02:00:26,424 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-06-07 02:00:26,531 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1020150863:84 , srv1022936965:84 , srv1029117043:84 , srv1034807491:84 , srv1100881370:84 , srv1108743038:84 , srv1116350659:84 , srv1134713911:84 , srv1156338532:84 , srv1171245716:84 , srv1193544616:84 , srv1226009976:84 , srv1237870602:84 , srv1260628716:84 , srv1275254950:84 , srv1278870524:84 , srv1299957826:84 , srv1344950133:84 , srv1345261581:84 , srv1353438238:84 , srv1353588939:84 , srv137679572:84 , srv1384105860:84 , srv1389630559:84 , srv1405558690:84 , srv1453167716:84 , srv148360184:84 , srv1512142173:84 , srv1514131280:84 , srv1529271630:84 , srv153260657:84 , srv1542227032:84 , srv1552775116:84 , srv157192428:84 , srv1576647180:84 , srv1601887939:84 , srv1667900078:84 , srv1680370394:84 , srv1700969786:84 , srv1729392629:84 , srv1737987627:84 , srv1787188381:84 , srv1789010588:84 , srv1795317828:84 , srv1799139534:84 , srv1800620892:84 , srv1821039314:84 , srv1851804346:84 , srv1861561118:84 , srv1870359121:84 , srv188425809:84 , srv1924667064:84 , srv196597995:84 , srv1966663568:84 , srv1988845170:84 , srv1992475289:84 , srv1999632681:84 , srv2014279354:84 , srv2025250192:84 , srv2035048178:84 , srv2097951428:84 , srv2099241466:84 , srv2100396178:84 , srv2106232240:84 , srv2135965631:84 , srv217772664:84 , srv237822384:84 , srv305266551:84 , srv308648761:84 , srv339566314:84 , srv359500098:84 , srv365217552:84 , srv382951458:84 , srv38389610:84 , srv415524644:84 , srv42767067:84 , srv516597753:84 , srv578885668:84 , srv606743363:84 , srv630680640:84 , srv643276303:84 , srv656535138:84 , srv657429979:84 , srv723071228:84 , srv729109090:84 , srv759969417:84 , srv771824459:84 , srv774374054:84 , srv787212763:84 , srv810526616:84 , srv852872571:84 , srv881308523:84 , srv896909735:84 , srv912081330:84 , srv945566264:84 , srv952616030:84 , srv992265303:84 , srv99414788:84 , srv997833234:84 , srv1589513336:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-06-07 02:00:26,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-06-07 02:00:26,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1667900078=37, srv339566314=70, srv1988845170=55, srv2035048178=60, srv1299957826=16, srv810526616=90, srv1237870602=12, srv1799139534=45, srv308648761=69, srv1022936965=1, srv1134713911=7, srv38389610=74, srv382951458=73, srv2135965631=65, srv1156338532=8, srv1529271630=29, srv305266551=68, srv1737987627=41, srv1870359121=50, srv42767067=76, srv1861561118=49, srv896909735=93, srv365217552=72, srv99414788=98, srv578885668=78, srv188425809=51, srv1700969786=39, srv1851804346=48, srv657429979=83, srv723071228=84, srv1787188381=42, srv643276303=81, srv196597995=53, srv852872571=91, srv1353438238=19, srv1680370394=38, srv787212763=89, srv912081330=94, srv1966663568=54, srv1116350659=6, srv1275254950=14, srv1029117043=2, srv1171245716=9, srv1100881370=4, srv1405558690=24, srv1795317828=44, srv137679572=21, srv1512142173=27, srv1601887939=36, srv759969417=86, srv2025250192=59, srv952616030=96, srv415524644=75, srv771824459=87, srv1034807491=3, srv606743363=79, srv1226009976=11, srv1924667064=52, srv1589513336=35, srv1453167716=25, srv1576647180=34, srv881308523=92, srv1345261581=18, srv516597753=77, srv2106232240=64, srv2014279354=58, srv2097951428=61, srv1992475289=56, srv1789010588=43, srv2099241466=62, srv2100396178=63, srv774374054=88, srv1514131280=28, srv217772664=66, srv630680640=80, srv1193544616=10, srv1278870524=15, srv1260628716=13, srv1344950133=17, srv1108743038=5, srv153260657=30, srv359500098=71, srv945566264=95, srv992265303=97, srv1821039314=47, srv237822384=67, srv1384105860=22, srv1389630559=23, srv1999632681=57, srv157192428=33, srv656535138=82, srv1542227032=31, srv1353588939=20, srv997833234=99, srv1729392629=40, srv1552775116=32, srv1800620892=46, srv1020150863=0, srv148360184=26, srv729109090=85} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-06-07 02:00:26,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-06-07 02:00:26,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-06-07 02:00:26,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-06-07 02:00:26,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-06-07 02:00:26,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-06-07 02:00:26,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-06-07 02:00:26,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-06-07 02:00:26,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-06-07 02:00:26,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-06-07 02:00:26,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-06-07 02:00:26,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-06-07 02:00:26,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-06-07 02:00:26,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-06-07 02:00:26,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-06-07 02:00:26,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-06-07 02:00:26,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-06-07 02:00:26,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-06-07 02:00:26,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-06-07 02:00:26,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-06-07 02:00:26,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-06-07 02:00:26,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-06-07 02:00:26,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-06-07 02:00:26,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-06-07 02:00:26,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-06-07 02:00:26,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-06-07 02:00:26,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-06-07 02:00:26,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-06-07 02:00:26,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-06-07 02:00:26,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-06-07 02:00:26,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-06-07 02:00:26,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-06-07 02:00:26,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-06-07 02:00:26,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-06-07 02:00:26,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-06-07 02:00:26,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-06-07 02:00:26,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-06-07 02:00:26,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-06-07 02:00:26,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-06-07 02:00:26,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-06-07 02:00:26,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-06-07 02:00:26,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-06-07 02:00:26,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-06-07 02:00:26,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-06-07 02:00:26,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-06-07 02:00:26,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-06-07 02:00:26,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-06-07 02:00:26,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-06-07 02:00:26,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-06-07 02:00:26,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-06-07 02:00:26,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-06-07 02:00:26,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-06-07 02:00:26,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-06-07 02:00:26,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-06-07 02:00:26,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-06-07 02:00:26,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-06-07 02:00:26,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-06-07 02:00:26,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-06-07 02:00:26,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-06-07 02:00:26,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-06-07 02:00:26,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-06-07 02:00:26,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-06-07 02:00:26,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-06-07 02:00:26,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-06-07 02:00:26,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-06-07 02:00:26,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-06-07 02:00:26,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-06-07 02:00:26,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-06-07 02:00:26,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-06-07 02:00:26,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-06-07 02:00:26,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-06-07 02:00:26,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-06-07 02:00:26,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-06-07 02:00:26,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-06-07 02:00:26,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-06-07 02:00:26,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-06-07 02:00:26,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-06-07 02:00:26,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-06-07 02:00:26,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-06-07 02:00:26,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-06-07 02:00:26,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-06-07 02:00:26,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-06-07 02:00:26,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-06-07 02:00:26,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-06-07 02:00:26,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-06-07 02:00:26,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-06-07 02:00:26,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-06-07 02:00:26,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-06-07 02:00:26,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-06-07 02:00:26,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-06-07 02:00:26,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-06-07 02:00:26,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-06-07 02:00:26,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-06-07 02:00:26,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-06-07 02:00:26,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-06-07 02:00:26,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-06-07 02:00:26,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-06-07 02:00:26,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-06-07 02:00:26,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-06-07 02:00:26,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-06-07 02:00:26,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-06-07 02:00:26,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-06-07 02:00:26,925 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-06-07 02:00:26,925 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-06-07 02:00:26,925 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-06-07 02:00:27,181 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 528 ms to try 5000 different iterations.  Found a solution that moves 4653 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18294857450962604. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03914187393379701, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055304483168960715, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.517, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03914187393379701, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17866666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24025, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-06-07 02:00:27,193 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1405558690:82 , srv1453167716:82 , srv2099241466:82 , srv42767067:82 , srv1353588939:83 , srv1729392629:83 , srv2035048178:83 , srv606743363:83 , srv723071228:83 , srv99414788:83 , srv1260628716:84 , srv1514131280:84 , srv1529271630:84 , srv188425809:84 , srv237822384:84 , srv305266551:84 , srv339566314:84 , srv787212763:84 , srv1034807491:85 , srv1171245716:85 , srv148360184:85 , srv157192428:85 , srv1667900078:85 , srv1795317828:85 , srv1861561118:85 , srv2025250192:85 , srv2106232240:85 , srv2135965631:85 , srv516597753:85 , srv643276303:85 , srv657429979:85 , srv952616030:85 , srv1029117043:86 , srv1100881370:86 , srv1108743038:86 , srv1116350659:86 , srv1156338532:86 , srv1275254950:86 , srv1299957826:86 , srv1512142173:86 , srv1552775116:86 , srv1680370394:86 , srv1700969786:86 , srv1787188381:86 , srv1789010588:86 , srv196597995:86 , srv1992475289:86 , srv2014279354:86 , srv308648761:86 , srv415524644:86 , srv578885668:86 , srv771824459:86 , srv852872571:86 , srv881308523:86 , srv997833234:86 , srv1020150863:87 , srv1193544616:87 , srv1345261581:87 , srv1353438238:87 , srv1389630559:87 , srv153260657:87 , srv1542227032:87 , srv1799139534:87 , srv1800620892:87 , srv1988845170:87 , srv2097951428:87 , srv2100396178:87 , srv365217552:87 , srv656535138:87 , srv810526616:87 , srv992265303:87 , srv1237870602:88 , srv1278870524:88 , srv1344950133:88 , srv1851804346:88 , srv1870359121:88 , srv1966663568:88 , srv217772664:88 , srv359500098:88 , srv759969417:88 , srv1022936965:89 , srv137679572:89 , srv1384105860:89 , srv1737987627:89 , srv38389610:89 , srv774374054:89 , srv1134713911:90 , srv1226009976:90 , srv1576647180:90 , srv382951458:90 , srv896909735:90 , srv912081330:90 , srv945566264:90 , srv1924667064:91 , srv1999632681:91 , srv630680640:91 , srv729109090:91 , srv1601887939:92 , srv1821039314:92 , srv1589513336:438 }
2023-06-07 02:00:27,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-06-07 02:00:27,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1667900078=37, srv339566314=70, srv1988845170=55, srv2035048178=60, srv1299957826=16, srv810526616=90, srv1237870602=12, srv1799139534=45, srv308648761=69, srv1022936965=1, srv1134713911=7, srv38389610=74, srv382951458=73, srv2135965631=65, srv1156338532=8, srv1529271630=29, srv305266551=68, srv1737987627=41, srv1870359121=50, srv42767067=76, srv1861561118=49, srv896909735=93, srv365217552=72, srv99414788=98, srv578885668=78, srv188425809=51, srv1700969786=39, srv1851804346=48, srv657429979=83, srv723071228=84, srv1787188381=42, srv643276303=81, srv196597995=53, srv852872571=91, srv1353438238=19, srv1680370394=38, srv787212763=89, srv912081330=94, srv1966663568=54, srv1116350659=6, srv1275254950=14, srv1029117043=2, srv1171245716=9, srv1100881370=4, srv1405558690=24, srv1795317828=44, srv137679572=21, srv1512142173=27, srv1601887939=36, srv759969417=86, srv2025250192=59, srv952616030=96, srv415524644=75, srv771824459=87, srv1034807491=3, srv606743363=79, srv1226009976=11, srv1924667064=52, srv1589513336=35, srv1453167716=25, srv1576647180=34, srv881308523=92, srv1345261581=18, srv516597753=77, srv2106232240=64, srv2014279354=58, srv2097951428=61, srv1992475289=56, srv1789010588=43, srv2099241466=62, srv2100396178=63, srv774374054=88, srv1514131280=28, srv217772664=66, srv630680640=80, srv1193544616=10, srv1278870524=15, srv1260628716=13, srv1344950133=17, srv1108743038=5, srv153260657=30, srv359500098=71, srv945566264=95, srv992265303=97, srv1821039314=47, srv237822384=67, srv1384105860=22, srv1389630559=23, srv1999632681=57, srv157192428=33, srv656535138=82, srv1542227032=31, srv1353588939=20, srv997833234=99, srv1729392629=40, srv1552775116=32, srv1800620892=46, srv1020150863=0, srv148360184=26, srv729109090=85} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-06-07 02:00:27,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-06-07 02:00:27,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-06-07 02:00:27,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-06-07 02:00:27,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-06-07 02:00:27,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-06-07 02:00:27,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-06-07 02:00:27,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-06-07 02:00:27,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-06-07 02:00:27,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-06-07 02:00:27,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-06-07 02:00:27,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-06-07 02:00:27,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-06-07 02:00:27,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-06-07 02:00:27,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-06-07 02:00:27,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-06-07 02:00:27,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-06-07 02:00:27,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-06-07 02:00:27,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-06-07 02:00:27,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-06-07 02:00:27,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-06-07 02:00:27,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-06-07 02:00:27,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-06-07 02:00:27,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-06-07 02:00:27,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-06-07 02:00:27,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-06-07 02:00:27,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-06-07 02:00:27,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-06-07 02:00:27,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-06-07 02:00:27,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-06-07 02:00:27,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-06-07 02:00:27,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-06-07 02:00:27,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-06-07 02:00:27,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-06-07 02:00:27,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-06-07 02:00:27,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-06-07 02:00:27,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-06-07 02:00:27,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-06-07 02:00:27,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-06-07 02:00:27,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-06-07 02:00:27,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-06-07 02:00:27,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-06-07 02:00:27,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-06-07 02:00:27,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-06-07 02:00:27,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-06-07 02:00:27,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-06-07 02:00:27,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-06-07 02:00:27,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-06-07 02:00:27,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-06-07 02:00:27,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-06-07 02:00:27,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-06-07 02:00:27,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-06-07 02:00:27,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-06-07 02:00:27,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-06-07 02:00:27,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-06-07 02:00:27,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-06-07 02:00:27,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-06-07 02:00:27,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-06-07 02:00:27,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-06-07 02:00:27,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-06-07 02:00:27,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-06-07 02:00:27,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-06-07 02:00:27,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-06-07 02:00:27,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-06-07 02:00:27,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-06-07 02:00:27,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-06-07 02:00:27,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-06-07 02:00:27,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-06-07 02:00:27,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-06-07 02:00:27,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-06-07 02:00:27,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-06-07 02:00:27,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-06-07 02:00:27,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-06-07 02:00:27,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-06-07 02:00:27,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-06-07 02:00:27,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-06-07 02:00:27,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-06-07 02:00:27,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-06-07 02:00:27,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-06-07 02:00:27,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-06-07 02:00:27,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-06-07 02:00:27,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-06-07 02:00:27,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-06-07 02:00:27,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-06-07 02:00:27,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-06-07 02:00:27,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-06-07 02:00:27,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-06-07 02:00:27,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-06-07 02:00:27,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-06-07 02:00:27,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-06-07 02:00:27,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-06-07 02:00:27,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-06-07 02:00:27,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-06-07 02:00:27,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-06-07 02:00:27,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-06-07 02:00:27,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-06-07 02:00:27,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-06-07 02:00:27,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-06-07 02:00:27,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-06-07 02:00:27,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-06-07 02:00:27,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-06-07 02:00:27,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-06-07 02:00:27,320 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03914187393379701, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055304483168960715, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03914187393379701, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17866666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24025, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-06-07 02:00:27,320 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-06-07 02:00:27,320 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1829159935072583, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03914187393379701, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055304483168960715, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03914187393379701, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17866666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24025, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-06-07 02:00:27,499 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 258 ms to try 5000 different iterations.  Found a solution that moves 2633 regions; Going from a computed imbalance of 0.1829159935072583 to a new imbalance of 0.014746115625390053. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020727509006864032); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03296769156111309, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29255555555555557, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020727509006864032); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0115); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04583333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-06-07 02:00:27,507 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1821039314:79 , srv630680640:79 , srv137679572:80 , srv1576647180:80 , srv1924667064:80 , srv382951458:80 , srv1134713911:81 , srv1601887939:81 , srv1022936965:82 , srv1226009976:82 , srv1999632681:82 , srv729109090:82 , srv1384105860:83 , srv1542227032:83 , srv1552775116:83 , srv1737987627:83 , srv1800620892:83 , srv1870359121:83 , srv217772664:83 , srv365217552:83 , srv38389610:83 , srv896909735:83 , srv1278870524:84 , srv1344950133:84 , srv1789010588:84 , srv1799139534:84 , srv1992475289:84 , srv2014279354:84 , srv2097951428:84 , srv415524644:84 , srv656535138:84 , srv774374054:84 , srv912081330:84 , srv992265303:84 , srv1100881370:85 , srv1108743038:85 , srv1193544616:85 , srv1237870602:85 , srv1389630559:85 , srv1680370394:85 , srv1787188381:85 , srv1861561118:85 , srv1988845170:85 , srv810526616:85 , srv997833234:85 , srv1029117043:86 , srv1034807491:86 , srv1353438238:86 , srv148360184:86 , srv1514131280:86 , srv153260657:86 , srv157192428:86 , srv308648761:86 , srv359500098:86 , srv643276303:86 , srv657429979:86 , srv852872571:86 , srv945566264:86 , srv1171245716:87 , srv1260628716:87 , srv1299957826:87 , srv1512142173:87 , srv1529271630:87 , srv1700969786:87 , srv1966663568:87 , srv2025250192:87 , srv2100396178:87 , srv2106232240:87 , srv516597753:87 , srv759969417:87 , srv771824459:87 , srv1020150863:88 , srv1116350659:88 , srv1275254950:88 , srv1345261581:88 , srv1667900078:88 , srv1729392629:88 , srv188425809:88 , srv196597995:88 , srv2135965631:88 , srv305266551:88 , srv578885668:88 , srv1156338532:89 , srv1353588939:89 , srv1795317828:89 , srv1851804346:89 , srv2099241466:89 , srv339566314:89 , srv606743363:89 , srv723071228:89 , srv952616030:89 , srv99414788:89 , srv1405558690:90 , srv1453167716:90 , srv237822384:90 , srv787212763:90 , srv881308523:90 , srv2035048178:91 , srv42767067:91 , srv1589513336:520 }
2023-06-07 02:00:27,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-06-07 02:00:27,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1667900078=37, srv339566314=70, srv1988845170=55, srv2035048178=60, srv1299957826=16, srv810526616=90, srv1237870602=12, srv1799139534=45, srv308648761=69, srv1022936965=1, srv1134713911=7, srv38389610=74, srv382951458=73, srv2135965631=65, srv1156338532=8, srv1529271630=29, srv305266551=68, srv1737987627=41, srv1870359121=50, srv42767067=76, srv1861561118=49, srv896909735=93, srv365217552=72, srv99414788=98, srv578885668=78, srv188425809=51, srv1700969786=39, srv1851804346=48, srv657429979=83, srv723071228=84, srv1787188381=42, srv643276303=81, srv196597995=53, srv852872571=91, srv1353438238=19, srv1680370394=38, srv787212763=89, srv912081330=94, srv1966663568=54, srv1116350659=6, srv1275254950=14, srv1029117043=2, srv1171245716=9, srv1100881370=4, srv1405558690=24, srv1795317828=44, srv137679572=21, srv1512142173=27, srv1601887939=36, srv759969417=86, srv2025250192=59, srv952616030=96, srv415524644=75, srv771824459=87, srv1034807491=3, srv606743363=79, srv1226009976=11, srv1924667064=52, srv1589513336=35, srv1453167716=25, srv1576647180=34, srv881308523=92, srv1345261581=18, srv516597753=77, srv2106232240=64, srv2014279354=58, srv2097951428=61, srv1992475289=56, srv1789010588=43, srv2099241466=62, srv2100396178=63, srv774374054=88, srv1514131280=28, srv217772664=66, srv630680640=80, srv1193544616=10, srv1278870524=15, srv1260628716=13, srv1344950133=17, srv1108743038=5, srv153260657=30, srv359500098=71, srv945566264=95, srv992265303=97, srv1821039314=47, srv237822384=67, srv1384105860=22, srv1389630559=23, srv1999632681=57, srv157192428=33, srv656535138=82, srv1542227032=31, srv1353588939=20, srv997833234=99, srv1729392629=40, srv1552775116=32, srv1800620892=46, srv1020150863=0, srv148360184=26, srv729109090=85} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-06-07 02:00:27,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-06-07 02:00:27,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-06-07 02:00:27,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-06-07 02:00:27,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-06-07 02:00:27,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-06-07 02:00:27,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-06-07 02:00:27,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-06-07 02:00:27,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-06-07 02:00:27,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-06-07 02:00:27,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-06-07 02:00:27,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-06-07 02:00:27,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-06-07 02:00:27,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-06-07 02:00:27,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-06-07 02:00:27,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-06-07 02:00:27,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-06-07 02:00:27,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-06-07 02:00:27,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-06-07 02:00:27,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-06-07 02:00:27,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-06-07 02:00:27,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-06-07 02:00:27,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-06-07 02:00:27,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-06-07 02:00:27,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-06-07 02:00:27,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-06-07 02:00:27,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-06-07 02:00:27,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-06-07 02:00:27,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-06-07 02:00:27,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-06-07 02:00:27,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-06-07 02:00:27,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-06-07 02:00:27,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-06-07 02:00:27,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-06-07 02:00:27,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-06-07 02:00:27,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-06-07 02:00:27,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-06-07 02:00:27,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-06-07 02:00:27,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-06-07 02:00:27,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-06-07 02:00:27,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-06-07 02:00:27,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-06-07 02:00:27,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-06-07 02:00:27,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-06-07 02:00:27,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-06-07 02:00:27,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-06-07 02:00:27,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-06-07 02:00:27,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-06-07 02:00:27,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-06-07 02:00:27,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-06-07 02:00:27,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-06-07 02:00:27,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-06-07 02:00:27,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-06-07 02:00:27,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-06-07 02:00:27,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-06-07 02:00:27,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-06-07 02:00:27,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-06-07 02:00:27,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-06-07 02:00:27,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-06-07 02:00:27,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-06-07 02:00:27,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-06-07 02:00:27,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-06-07 02:00:27,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-06-07 02:00:27,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-06-07 02:00:27,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-06-07 02:00:27,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-06-07 02:00:27,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-06-07 02:00:27,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-06-07 02:00:27,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-06-07 02:00:27,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-06-07 02:00:27,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-06-07 02:00:27,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-06-07 02:00:27,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-06-07 02:00:27,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-06-07 02:00:27,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-06-07 02:00:27,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-06-07 02:00:27,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-06-07 02:00:27,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-06-07 02:00:27,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-06-07 02:00:27,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-06-07 02:00:27,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-06-07 02:00:27,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-06-07 02:00:27,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-06-07 02:00:27,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-06-07 02:00:27,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-06-07 02:00:27,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-06-07 02:00:27,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-06-07 02:00:27,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-06-07 02:00:27,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-06-07 02:00:27,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-06-07 02:00:27,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-06-07 02:00:27,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-06-07 02:00:27,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-06-07 02:00:27,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-06-07 02:00:27,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-06-07 02:00:27,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-06-07 02:00:27,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-06-07 02:00:27,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-06-07 02:00:27,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-06-07 02:00:27,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-06-07 02:00:27,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-06-07 02:00:27,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-06-07 02:00:27,622 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020727509006864032); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03296769156111309, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020727509006864032); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0115); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04583333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-06-07 02:00:27,622 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-06-07 02:00:27,622 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014727678965335417, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020727509006864032); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03296769156111309, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020727509006864032); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0115); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04583333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-06-07 02:00:27,793 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 239 ms to try 5000 different iterations.  Found a solution that moves 1015 regions; Going from a computed imbalance of 0.014727678965335417 to a new imbalance of 7.599149896850558E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007562352524690594); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010814215194852728); MoveCostFunction : (multiplier=7.0, imbalance=0.11277777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007562352524690594); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-06-07 02:00:27,795 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1851804346:81 , srv1020150863:82 , srv237822384:82 , srv881308523:82 , srv945566264:82 , srv1275254950:83 , srv1345261581:83 , srv1353438238:83 , srv1667900078:83 , srv1795317828:83 , srv196597995:83 , srv1966663568:83 , srv2035048178:83 , srv359500098:83 , srv42767067:83 , srv787212763:83 , srv952616030:83 , srv1116350659:84 , srv1156338532:84 , srv1226009976:84 , srv1389630559:84 , srv1453167716:84 , srv1512142173:84 , srv1700969786:84 , srv1999632681:84 , srv2100396178:84 , srv2135965631:84 , srv308648761:84 , srv339566314:84 , srv578885668:84 , srv606743363:84 , srv759969417:84 , srv912081330:84 , srv1237870602:85 , srv1278870524:85 , srv1299957826:85 , srv1405558690:85 , srv1529271630:85 , srv153260657:85 , srv1601887939:85 , srv1737987627:85 , srv1988845170:85 , srv2025250192:85 , srv2106232240:85 , srv516597753:85 , srv729109090:85 , srv771824459:85 , srv774374054:85 , srv852872571:85 , srv896909735:85 , srv99414788:85 , srv997833234:85 , srv1029117043:86 , srv1134713911:86 , srv1171245716:86 , srv1193544616:86 , srv1260628716:86 , srv1344950133:86 , srv1353588939:86 , srv1384105860:86 , srv1542227032:86 , srv157192428:86 , srv1680370394:86 , srv1729392629:86 , srv1787188381:86 , srv1799139534:86 , srv1861561118:86 , srv1870359121:86 , srv188425809:86 , srv1924667064:86 , srv2097951428:86 , srv305266551:86 , srv38389610:86 , srv643276303:86 , srv723071228:86 , srv810526616:86 , srv992265303:86 , srv1022936965:87 , srv1034807491:87 , srv1100881370:87 , srv1108743038:87 , srv148360184:87 , srv1514131280:87 , srv1789010588:87 , srv1821039314:87 , srv1992475289:87 , srv2014279354:87 , srv2099241466:87 , srv217772664:87 , srv365217552:87 , srv382951458:87 , srv415524644:87 , srv630680640:87 , srv656535138:87 , srv657429979:87 , srv137679572:88 , srv1576647180:88 , srv1800620892:88 , srv1552775116:89 , srv1589513336:567 }
2023-06-07 02:00:27,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-06-07 02:00:27,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1667900078=37, srv339566314=70, srv1988845170=55, srv2035048178=60, srv1299957826=16, srv810526616=90, srv1237870602=12, srv1799139534=45, srv308648761=69, srv1022936965=1, srv1134713911=7, srv38389610=74, srv382951458=73, srv2135965631=65, srv1156338532=8, srv1529271630=29, srv305266551=68, srv1737987627=41, srv1870359121=50, srv42767067=76, srv1861561118=49, srv896909735=93, srv365217552=72, srv99414788=98, srv578885668=78, srv188425809=51, srv1700969786=39, srv1851804346=48, srv657429979=83, srv723071228=84, srv1787188381=42, srv643276303=81, srv196597995=53, srv852872571=91, srv1353438238=19, srv1680370394=38, srv787212763=89, srv912081330=94, srv1966663568=54, srv1116350659=6, srv1275254950=14, srv1029117043=2, srv1171245716=9, srv1100881370=4, srv1405558690=24, srv1795317828=44, srv137679572=21, srv1512142173=27, srv1601887939=36, srv759969417=86, srv2025250192=59, srv952616030=96, srv415524644=75, srv771824459=87, srv1034807491=3, srv606743363=79, srv1226009976=11, srv1924667064=52, srv1589513336=35, srv1453167716=25, srv1576647180=34, srv881308523=92, srv1345261581=18, srv516597753=77, srv2106232240=64, srv2014279354=58, srv2097951428=61, srv1992475289=56, srv1789010588=43, srv2099241466=62, srv2100396178=63, srv774374054=88, srv1514131280=28, srv217772664=66, srv630680640=80, srv1193544616=10, srv1278870524=15, srv1260628716=13, srv1344950133=17, srv1108743038=5, srv153260657=30, srv359500098=71, srv945566264=95, srv992265303=97, srv1821039314=47, srv237822384=67, srv1384105860=22, srv1389630559=23, srv1999632681=57, srv157192428=33, srv656535138=82, srv1542227032=31, srv1353588939=20, srv997833234=99, srv1729392629=40, srv1552775116=32, srv1800620892=46, srv1020150863=0, srv148360184=26, srv729109090=85} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-06-07 02:00:27,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-06-07 02:00:27,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-06-07 02:00:27,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-06-07 02:00:27,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-06-07 02:00:27,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-06-07 02:00:27,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-06-07 02:00:27,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-06-07 02:00:27,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-06-07 02:00:27,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-06-07 02:00:27,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-06-07 02:00:27,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-06-07 02:00:27,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-06-07 02:00:27,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-06-07 02:00:27,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-06-07 02:00:27,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-06-07 02:00:27,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-06-07 02:00:27,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-06-07 02:00:27,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-06-07 02:00:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-06-07 02:00:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-06-07 02:00:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-06-07 02:00:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-06-07 02:00:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-06-07 02:00:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-06-07 02:00:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-06-07 02:00:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-06-07 02:00:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-06-07 02:00:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-06-07 02:00:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-06-07 02:00:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-06-07 02:00:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-06-07 02:00:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-06-07 02:00:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-06-07 02:00:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-06-07 02:00:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-06-07 02:00:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-06-07 02:00:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-06-07 02:00:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-06-07 02:00:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-06-07 02:00:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-06-07 02:00:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-06-07 02:00:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-06-07 02:00:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-06-07 02:00:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-06-07 02:00:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-06-07 02:00:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-06-07 02:00:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-06-07 02:00:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-06-07 02:00:27,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-06-07 02:00:27,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-06-07 02:00:27,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-06-07 02:00:27,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-06-07 02:00:27,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-06-07 02:00:27,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-06-07 02:00:27,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-06-07 02:00:27,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-06-07 02:00:27,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-06-07 02:00:27,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-06-07 02:00:27,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-06-07 02:00:27,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-06-07 02:00:27,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-06-07 02:00:27,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-06-07 02:00:27,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-06-07 02:00:27,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-06-07 02:00:27,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-06-07 02:00:27,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-06-07 02:00:27,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-06-07 02:00:27,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-06-07 02:00:27,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-06-07 02:00:27,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-06-07 02:00:27,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-06-07 02:00:27,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-06-07 02:00:27,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-06-07 02:00:27,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-06-07 02:00:27,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-06-07 02:00:27,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-06-07 02:00:27,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-06-07 02:00:27,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-06-07 02:00:27,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-06-07 02:00:27,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-06-07 02:00:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-06-07 02:00:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-06-07 02:00:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-06-07 02:00:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-06-07 02:00:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-06-07 02:00:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-06-07 02:00:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-06-07 02:00:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-06-07 02:00:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-06-07 02:00:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-06-07 02:00:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-06-07 02:00:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-06-07 02:00:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-06-07 02:00:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-06-07 02:00:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-06-07 02:00:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-06-07 02:00:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-06-07 02:00:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-06-07 02:00:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-06-07 02:00:27,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-06-07 02:00:27,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-06-07 02:00:27,891 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007562352524690594); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010814215194852728); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007562352524690594); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-06-07 02:00:27,891 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-06-07 02:00:27,892 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.52807807780211E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007562352524690594); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010814215194852728); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007562352524690594); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-06-07 02:00:28,116 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 295 ms to try 5000 different iterations.  Found a solution that moves 301 regions; Going from a computed imbalance of 7.52807807780211E-4 to a new imbalance of 2.0153086985445646E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.03344444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-06-07 02:00:28,117 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1156338532:83 , srv148360184:83 , srv1022936965:84 , srv1029117043:84 , srv1034807491:84 , srv1100881370:84 , srv1108743038:84 , srv1116350659:84 , srv1171245716:84 , srv1193544616:84 , srv1237870602:84 , srv1278870524:84 , srv1299957826:84 , srv1344950133:84 , srv1345261581:84 , srv1353588939:84 , srv1384105860:84 , srv153260657:84 , srv1552775116:84 , srv1576647180:84 , srv1789010588:84 , srv1799139534:84 , srv1800620892:84 , srv1821039314:84 , srv1851804346:84 , srv188425809:84 , srv1966663568:84 , srv2100396178:84 , srv217772664:84 , srv305266551:84 , srv38389610:84 , srv578885668:84 , srv657429979:84 , srv723071228:84 , srv729109090:84 , srv759969417:84 , srv771824459:84 , srv774374054:84 , srv810526616:84 , srv852872571:84 , srv881308523:84 , srv896909735:84 , srv1020150863:85 , srv1134713911:85 , srv1226009976:85 , srv1260628716:85 , srv1275254950:85 , srv137679572:85 , srv1389630559:85 , srv1405558690:85 , srv1512142173:85 , srv1514131280:85 , srv157192428:85 , srv1601887939:85 , srv1700969786:85 , srv1729392629:85 , srv1737987627:85 , srv1787188381:85 , srv1795317828:85 , srv1870359121:85 , srv1924667064:85 , srv196597995:85 , srv1988845170:85 , srv1992475289:85 , srv1999632681:85 , srv2014279354:85 , srv2025250192:85 , srv2035048178:85 , srv2097951428:85 , srv2099241466:85 , srv2106232240:85 , srv2135965631:85 , srv339566314:85 , srv359500098:85 , srv365217552:85 , srv382951458:85 , srv415524644:85 , srv516597753:85 , srv630680640:85 , srv643276303:85 , srv656535138:85 , srv787212763:85 , srv912081330:85 , srv945566264:85 , srv952616030:85 , srv992265303:85 , srv99414788:85 , srv1353438238:86 , srv1453167716:86 , srv1529271630:86 , srv1542227032:86 , srv1667900078:86 , srv1680370394:86 , srv1861561118:86 , srv237822384:86 , srv308648761:86 , srv42767067:86 , srv606743363:86 , srv997833234:86 , srv1589513336:617 }
2023-06-07 02:00:28,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-06-07 02:00:28,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1667900078=37, srv339566314=70, srv1988845170=55, srv2035048178=60, srv1299957826=16, srv810526616=90, srv1237870602=12, srv1799139534=45, srv308648761=69, srv1022936965=1, srv1134713911=7, srv38389610=74, srv382951458=73, srv2135965631=65, srv1156338532=8, srv1529271630=29, srv305266551=68, srv1737987627=41, srv1870359121=50, srv42767067=76, srv1861561118=49, srv896909735=93, srv365217552=72, srv99414788=98, srv578885668=78, srv188425809=51, srv1700969786=39, srv1851804346=48, srv657429979=83, srv723071228=84, srv1787188381=42, srv643276303=81, srv196597995=53, srv852872571=91, srv1353438238=19, srv1680370394=38, srv787212763=89, srv912081330=94, srv1966663568=54, srv1116350659=6, srv1275254950=14, srv1029117043=2, srv1171245716=9, srv1100881370=4, srv1405558690=24, srv1795317828=44, srv137679572=21, srv1512142173=27, srv1601887939=36, srv759969417=86, srv2025250192=59, srv952616030=96, srv415524644=75, srv771824459=87, srv1034807491=3, srv606743363=79, srv1226009976=11, srv1924667064=52, srv1589513336=35, srv1453167716=25, srv1576647180=34, srv881308523=92, srv1345261581=18, srv516597753=77, srv2106232240=64, srv2014279354=58, srv2097951428=61, srv1992475289=56, srv1789010588=43, srv2099241466=62, srv2100396178=63, srv774374054=88, srv1514131280=28, srv217772664=66, srv630680640=80, srv1193544616=10, srv1278870524=15, srv1260628716=13, srv1344950133=17, srv1108743038=5, srv153260657=30, srv359500098=71, srv945566264=95, srv992265303=97, srv1821039314=47, srv237822384=67, srv1384105860=22, srv1389630559=23, srv1999632681=57, srv157192428=33, srv656535138=82, srv1542227032=31, srv1353588939=20, srv997833234=99, srv1729392629=40, srv1552775116=32, srv1800620892=46, srv1020150863=0, srv148360184=26, srv729109090=85} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-06-07 02:00:28,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-06-07 02:00:28,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-06-07 02:00:28,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-06-07 02:00:28,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-06-07 02:00:28,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-06-07 02:00:28,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-06-07 02:00:28,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-06-07 02:00:28,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-06-07 02:00:28,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-06-07 02:00:28,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-06-07 02:00:28,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-06-07 02:00:28,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-06-07 02:00:28,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-06-07 02:00:28,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-06-07 02:00:28,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-06-07 02:00:28,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-06-07 02:00:28,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-06-07 02:00:28,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-06-07 02:00:28,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-06-07 02:00:28,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-06-07 02:00:28,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-06-07 02:00:28,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-06-07 02:00:28,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-06-07 02:00:28,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-06-07 02:00:28,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-06-07 02:00:28,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-06-07 02:00:28,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-06-07 02:00:28,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-06-07 02:00:28,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-06-07 02:00:28,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-06-07 02:00:28,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-06-07 02:00:28,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-06-07 02:00:28,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-06-07 02:00:28,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-06-07 02:00:28,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-06-07 02:00:28,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-06-07 02:00:28,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-06-07 02:00:28,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-06-07 02:00:28,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-06-07 02:00:28,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-06-07 02:00:28,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-06-07 02:00:28,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-06-07 02:00:28,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-06-07 02:00:28,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-06-07 02:00:28,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-06-07 02:00:28,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-06-07 02:00:28,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-06-07 02:00:28,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-06-07 02:00:28,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-06-07 02:00:28,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-06-07 02:00:28,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-06-07 02:00:28,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-06-07 02:00:28,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-06-07 02:00:28,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-06-07 02:00:28,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-06-07 02:00:28,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-06-07 02:00:28,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-06-07 02:00:28,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-06-07 02:00:28,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-06-07 02:00:28,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-06-07 02:00:28,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-06-07 02:00:28,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-06-07 02:00:28,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-06-07 02:00:28,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-06-07 02:00:28,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-06-07 02:00:28,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-06-07 02:00:28,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-06-07 02:00:28,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-06-07 02:00:28,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-06-07 02:00:28,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-06-07 02:00:28,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-06-07 02:00:28,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-06-07 02:00:28,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-06-07 02:00:28,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-06-07 02:00:28,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-06-07 02:00:28,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-06-07 02:00:28,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-06-07 02:00:28,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-06-07 02:00:28,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-06-07 02:00:28,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-06-07 02:00:28,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-06-07 02:00:28,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-06-07 02:00:28,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-06-07 02:00:28,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-06-07 02:00:28,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-06-07 02:00:28,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-06-07 02:00:28,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-06-07 02:00:28,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-06-07 02:00:28,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-06-07 02:00:28,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-06-07 02:00:28,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-06-07 02:00:28,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-06-07 02:00:28,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-06-07 02:00:28,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-06-07 02:00:28,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-06-07 02:00:28,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-06-07 02:00:28,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-06-07 02:00:28,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-06-07 02:00:28,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-06-07 02:00:28,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-06-07 02:00:28,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-06-07 02:00:28,222 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.804543993780202E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-06-07 02:00:28,222 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1156338532:83 , srv148360184:83 , srv1022936965:84 , srv1029117043:84 , srv1034807491:84 , srv1100881370:84 , srv1108743038:84 , srv1116350659:84 , srv1171245716:84 , srv1193544616:84 , srv1237870602:84 , srv1278870524:84 , srv1299957826:84 , srv1344950133:84 , srv1345261581:84 , srv1353588939:84 , srv1384105860:84 , srv153260657:84 , srv1552775116:84 , srv1576647180:84 , srv1789010588:84 , srv1799139534:84 , srv1800620892:84 , srv1821039314:84 , srv1851804346:84 , srv188425809:84 , srv1966663568:84 , srv2100396178:84 , srv217772664:84 , srv305266551:84 , srv38389610:84 , srv578885668:84 , srv657429979:84 , srv723071228:84 , srv729109090:84 , srv759969417:84 , srv771824459:84 , srv774374054:84 , srv810526616:84 , srv852872571:84 , srv881308523:84 , srv896909735:84 , srv1020150863:85 , srv1134713911:85 , srv1226009976:85 , srv1260628716:85 , srv1275254950:85 , srv137679572:85 , srv1389630559:85 , srv1405558690:85 , srv1512142173:85 , srv1514131280:85 , srv157192428:85 , srv1601887939:85 , srv1700969786:85 , srv1729392629:85 , srv1737987627:85 , srv1787188381:85 , srv1795317828:85 , srv1870359121:85 , srv1924667064:85 , srv196597995:85 , srv1988845170:85 , srv1992475289:85 , srv1999632681:85 , srv2014279354:85 , srv2025250192:85 , srv2035048178:85 , srv2097951428:85 , srv2099241466:85 , srv2106232240:85 , srv2135965631:85 , srv339566314:85 , srv359500098:85 , srv365217552:85 , srv382951458:85 , srv415524644:85 , srv516597753:85 , srv630680640:85 , srv643276303:85 , srv656535138:85 , srv787212763:85 , srv912081330:85 , srv945566264:85 , srv952616030:85 , srv992265303:85 , srv99414788:85 , srv1353438238:86 , srv1453167716:86 , srv1529271630:86 , srv1542227032:86 , srv1667900078:86 , srv1680370394:86 , srv1861561118:86 , srv237822384:86 , srv308648761:86 , srv42767067:86 , srv606743363:86 , srv997833234:86 , srv1589513336:617 }
2023-06-07 02:00:28,249 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=147 (was 134) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 169), AvailableMemoryMB=5179 (was 5658)
2023-06-07 02:00:28,257 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=147, ProcessCount=167, AvailableMemoryMB=5179
2023-06-07 02:00:28,258 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-06-07 02:00:28,258 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-06-07 02:00:28,265 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1044742028:84 , srv1073557623:84 , srv1091835362:84 , srv1105067612:84 , srv1331507720:84 , srv1386388574:84 , srv1466683360:84 , srv1559497848:84 , srv1626349324:84 , srv1674369078:84 , srv170072534:84 , srv1715830085:84 , srv1729647856:84 , srv1791148144:84 , srv1792037979:84 , srv1860434300:84 , srv1909233321:84 , srv1909262341:84 , srv2000362045:84 , srv2076402512:84 , srv231319831:84 , srv243510973:84 , srv483285489:84 , srv503164301:84 , srv550980162:84 , srv600629875:84 , srv743598:84 , srv761631502:84 , srv911526178:84 , srv828490442:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-06-07 02:00:28,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-06-07 02:00:28,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2076402512=19, srv1105067612=3, srv231319831=20, srv911526178=29, srv1073557623=1, srv1674369078=9, srv600629875=25, srv1386388574=5, srv1466683360=6, srv1729647856=12, srv2000362045=18, srv1331507720=4, srv1559497848=7, srv1860434300=15, srv1909262341=17, srv1044742028=0, srv1715830085=11, srv743598=26, srv1909233321=16, srv503164301=23, srv243510973=21, srv828490442=28, srv1792037979=14, srv761631502=27, srv1091835362=2, srv1626349324=8, srv170072534=10, srv550980162=24, srv483285489=22, srv1791148144=13} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-06-07 02:00:28,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-06-07 02:00:28,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-06-07 02:00:28,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-06-07 02:00:28,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-06-07 02:00:28,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-06-07 02:00:28,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-06-07 02:00:28,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-06-07 02:00:28,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-06-07 02:00:28,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-06-07 02:00:28,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-06-07 02:00:28,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-06-07 02:00:28,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-06-07 02:00:28,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-06-07 02:00:28,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-06-07 02:00:28,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-06-07 02:00:28,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-06-07 02:00:28,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-06-07 02:00:28,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-06-07 02:00:28,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-06-07 02:00:28,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-06-07 02:00:28,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-06-07 02:00:28,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-06-07 02:00:28,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-06-07 02:00:28,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-06-07 02:00:28,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-06-07 02:00:28,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-06-07 02:00:28,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-06-07 02:00:28,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-06-07 02:00:28,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-06-07 02:00:28,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-06-07 02:00:28,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-06-07 02:00:28,293 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1122640980663525, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-06-07 02:00:28,293 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912546783771526, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1122640980663525, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-06-07 02:02:28,283 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9819446 different iterations.  Found a solution that moves 1836 regions; Going from a computed imbalance of 0.9912546783771526 to a new imbalance of 4.2853155918867096E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.68, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-06-07 02:02:28,289 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1044742028:90 , srv1073557623:90 , srv1091835362:90 , srv1105067612:90 , srv1331507720:90 , srv1386388574:90 , srv1466683360:90 , srv1559497848:90 , srv1626349324:90 , srv1674369078:90 , srv170072534:90 , srv1715830085:90 , srv1729647856:90 , srv1791148144:90 , srv1792037979:90 , srv1860434300:90 , srv1909233321:90 , srv1909262341:90 , srv2000362045:90 , srv2076402512:90 , srv231319831:90 , srv243510973:90 , srv483285489:90 , srv503164301:90 , srv550980162:90 , srv600629875:90 , srv743598:90 , srv761631502:90 , srv828490442:90 , srv911526178:90 }
2023-06-07 02:02:28,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-06-07 02:02:28,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2076402512=19, srv1105067612=3, srv231319831=20, srv911526178=29, srv1073557623=1, srv1674369078=9, srv600629875=25, srv1386388574=5, srv1466683360=6, srv1729647856=12, srv2000362045=18, srv1331507720=4, srv1559497848=7, srv1860434300=15, srv1909262341=17, srv1044742028=0, srv1715830085=11, srv743598=26, srv1909233321=16, srv503164301=23, srv243510973=21, srv828490442=28, srv1792037979=14, srv761631502=27, srv1091835362=2, srv1626349324=8, srv170072534=10, srv550980162=24, srv483285489=22, srv1791148144=13} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-06-07 02:02:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-06-07 02:02:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-06-07 02:02:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-06-07 02:02:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-06-07 02:02:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-06-07 02:02:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-06-07 02:02:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-06-07 02:02:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-06-07 02:02:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-06-07 02:02:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-06-07 02:02:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-06-07 02:02:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-06-07 02:02:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-06-07 02:02:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-06-07 02:02:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-06-07 02:02:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-06-07 02:02:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-06-07 02:02:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-06-07 02:02:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-06-07 02:02:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-06-07 02:02:28,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-06-07 02:02:28,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-06-07 02:02:28,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-06-07 02:02:28,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-06-07 02:02:28,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-06-07 02:02:28,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-06-07 02:02:28,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-06-07 02:02:28,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-06-07 02:02:28,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-06-07 02:02:28,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-06-07 02:02:28,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-06-07 02:02:28,304 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-06-07 02:02:28,304 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1044742028:90 , srv1073557623:90 , srv1091835362:90 , srv1105067612:90 , srv1331507720:90 , srv1386388574:90 , srv1466683360:90 , srv1559497848:90 , srv1626349324:90 , srv1674369078:90 , srv170072534:90 , srv1715830085:90 , srv1729647856:90 , srv1791148144:90 , srv1792037979:90 , srv1860434300:90 , srv1909233321:90 , srv1909262341:90 , srv2000362045:90 , srv2076402512:90 , srv231319831:90 , srv243510973:90 , srv483285489:90 , srv503164301:90 , srv550980162:90 , srv600629875:90 , srv743598:90 , srv761631502:90 , srv828490442:90 , srv911526178:90 }
2023-06-07 02:02:28,315 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=140 (was 147), ProcessCount=165 (was 167), AvailableMemoryMB=5054 (was 5179)